In 2026, Building AI Search Means Building Agent Memory: Insights from Elastic's Xiao Han

Forum topic · ✨步子哥 · 2026-04-25

Summary

At the Elastic China AI Search Conference in Beijing on April 18, Elastic global VP Xiao Han (former Jina AI founder/CEO) argued that in 2026, doing AI search is essentially doing agent memory. He traced search's evolution from keyword retrieval (BM25) to vector search, hybrid retrieval, RAG, Deep Research, and now long-horizon autonomous agents, where persistent memory becomes a hard requirement. Using a real failure case (an agent misinterpreting "911" as September 11 instead of the Porsche model), he showed that memory retrieval often fails at query construction. He outlined three camps of memory products—database-based, file-based (e.g., OpenClaw), and model-based (e.g., Letta)—identified selective forgetting as the field's central unsolved problem (citing Andrej Karpathy), and reviewed benchmarks like LongMemEval and MemoryArena. He predicted unlearning will be repaid as technical debt and that recommendation-system companies may rise again in the agent memory era.

In 2026, Building AI Search Means Building Agent Memory

On April 18, at the Elastic China AI Search Conference in Beijing, Xiao Han, Global VP at Elastic and former founder/CEO of Jina AI, delivered a keynote on why, in 2026, building AI search is essentially building agent memory systems.

Key points

Search has evolved through distinct phases: keyword retrieval (BM25/TF-IDF, Lucene) around 2009; vector search (Faiss, Milvus) after 2015; hybrid keyword+vector search; BERT-based encoders after 2018; RAG after ChatGPT in 2022; "Deep Research" combining reasoning with search in 2025 (e.g., DeepSeek-R1); and in 2026, agent platforms (e.g., "OpenClaw") that run autonomously for hours.
The 2026 technical theme is Long Horizon Tasks: when agents run 4–5 hours or more without human intervention, persistent memory shifts from optional to essential. Search's goal changes from "giving users ten results" to "giving an agent the right answer."
Per Andrej Karpathy, the fundamental flaw of all current LLM personalization is the inability to selectively forget.

A real-world memory failure case

Xiao Han asked an agent (using top-tier Claude Opus 4.6) to find a "911 chart" he had previously made—referring to the Porsche 911. The agent translated "911" into "September 11th," searched, and found nothing. Debugging showed the retrieval pipeline (Grep, vector search, traversing Memory Markdown, session logs, file system) was methodologically sound; the failure was in query construction. Using the model code "992.2" immediately located the chart. Lesson: even with a complete retrieval chain, a wrong query sinks everything.

Core problems: representation and forgetting

Representation: Should memory be stored as chat logs, fact triples, knowledge graphs, vectors, timestamp-event pairs, or model weights? Each choice yields a very different system. Additional questions: how to do CRUD on this representation, how to extract memory without breaking conversation flow, and can memory transfer across models? Xiao argues memory should be decoupled from the model layer.
Selective forgetting: Mapping biology to AI—hippocampus as RAG-style real-time retrieval, neocortex as fine-tuning/pretraining, prefrontal cortex as the context window—the key question is *when to forget*. Current systems only accumulate. Three defects plague memory today: omission, distortion (hallucination), and hallucination inheritance. This creates a cold-start paradox: new agents feel great, then degrade over time as memories accumulate noise, eroding user trust.

Three camps of memory products

Database camp: vector DBs, SQL, key-value stores; high structure and query efficiency, but schema-locked.
File camp: Markdown/text files as the source of truth; transparent, editable, versionable, but files bloat without smart forgetting. Examples: OpenClaw ("lobster") and MemSearch.
Model camp: truth lives in model weights or context; zero-config and adaptive, but a black box. Examples: Letta and ChatGPT.

Consensus workflow

1. Extraction: LLM extracts facts from chat history. 2. Storage: vectors, knowledge graphs, or temporal databases (time matters for forgetting policies). 3. Hybrid retrieval: with RRF fusion, MMR deduplication, and time-decay ranking. 4. Conflict resolution and decay: handling contradictory memories and weighting old ones down.

Case study: OpenClaw's file-based memory

OpenClaw uses daily Markdown journals plus memory.md (long-term user profile) and soul.md (agent meta-info) as truth sources, deriving indexes in SQLite FTS5 (BM25) and LanceDB (vectors), with RRF/MMR/time-decay hybrid retrieval exposed via memory_search and memory_get. It simulates human sleep stages (deep, light, REM) with thresholds deciding which memories get promoted to long-term—mimicking forgetting.

The model-based direction

Xiao personally favors the model camp: perhaps Transformers themselves will solve memory—a "Bitter Lesson" outcome. Context windows keep growing: Qwen 3.5 MoE scales to 1M tokens on a 24GB GPU, and EverMind AI has pushed context to 100M tokens with Memory Sparse Attention (MSA). If all agent memory fits in context, hierarchical designs may become unnecessary.

Benchmarks and outlook

Benchmarks cited: LongMemEval (500 questions across extraction, multi-session reasoning, knowledge updating, temporal reasoning, safety refusal), MABench (ICLR 2026), MemoryArena, EverMemBench (first multi-party scenario, 1M+ tokens), and Tsinghua's MemoryBench. Findings: longer contexts degrade memory; multi-session reasoning and conflict resolution remain very hard. EverMind AI raises a novel question: in group chats, whose information should the agent remember, and how?

Predictions

1. No unified paradigm yet—the field resembles deep learning before 2017, with competing architectures. 2. Pure text-based approaches are giving up—Grep-only systems have an obvious ceiling. 3. Unlearning is long-overdue technical debt—machine unlearning has been discussed since 2009 with no perfect solution; polluted recommendation profiles that can't be reset share the same root cause. 4. Recommendation systems will resurge—agent memory is personalization (preferences, behavior, history), exactly what companies like Xiaohongshu, Douyin, and Taobao have done for two decades.

The core thesis: in 2026, building AI search is building agent memory. Memory is the critical bottleneck determining whether agents can run continuously, efficiently, and reliably—whoever solves unified paradigms, forgetting, and cross-model transfer will lead in the AI 2.0 era.