Have you ever chatted with an AI assistant for three days, then asked on day four what coffee flavor you mentioned liking last week—only to watch it scan the entire chat history, burning a large amount of tokens in the process?
Every time it needs to "remember," it invokes a large model to summarize, extract, and retrieve past conversations. As memory accumulates, the cost of remembering grows too. It's like re-reading ten years of diaries and writing a summary every time you want to recall one event—recollection alone exhausts your energy.
Zero-Mem asks a radical question: do memory operations really need to call a large language model?
Paper: https://arxiv.org/abs/2607.29377 Code: https://github.com/TheMoon0815/Zero-mem
The "Generation Tax" in Existing Agent Memory Systems
To see why Zero-Mem matters, look at the problem with current systems.
Mainstream agent memory systems—Zep, Mem0, A-Mem, MemoryOS, GAM—all use LLMs to manage memory. Specifically, they delegate to the LLM:
1. Summarization: compressing long conversations into key points 2. Extraction: pulling entities, relations, and events from dialogue 3. Update: merging, modifying, deleting old memories when new info arrives 4. Retrieval: using the LLM to decide which memories are relevant
Each operation is an LLM call, costing input + output tokens. A long conversation with hundreds of turns may trigger dozens of memory operations. That's the "generation tax": you pay not only for the final answer but for remembering itself.
The deeper problem: generation means loss. Compressing dialogue into summaries loses detail. Merging conversations blurs timelines. LLM "understanding" of memory can introduce hallucinations. Original evidence gets buried under layers of abstraction, breaking the provenance chain for final answers.
Zero-Mem's Core Insight: Don't Generate, Just Filter
> Memory operations need not generate—they only need structured filtering.
It's like library management. Traditional approach: every new batch of books gets read by a librarian (the LLM), who writes summaries, categorizes, and builds index cards. Zero-Mem's approach: books go on shelves untouched; only a catalog system (entity graph + temporal hierarchy) is built, and lookups go through the catalog straight to the original book.
Concretely, Zero-Mem keeps all raw interaction traces without any generative abstraction, and builds two complementary "views" over them.
View 1: Entity-Context Graph (relational view)
A lightweight NER model (e.g., spaCy—not an LLM) scans each conversation to extract person, place, and organization names, then builds a graph:
- Edges between entity nodes and context nodes: an entity appeared in a given dialogue segment
- Edges between context nodes: adjacency between dialogue segments
- Turn: one complete interaction
- Window: a few adjacent turns
- Episode: several windows forming one event
- Local: immediate context around a candidate evidence item
- "Xiao Wang's project progress" → relational → graph view first
- "What did we discuss yesterday afternoon" → temporal → hierarchy view first
- "Xiao Wang said the project was delayed; how was it resolved?" → hybrid → both views
- LLM calls for memory operations: 0 (vs. dozens for baselines)
- LLM tokens for memory operations: 0 (encoder compute is billed separately, not as LLM tokens)
- 57.6% latency reduction for memory operations vs. the fastest baseline (same final QA reader, equal context budget)
- Competitive performance (not top, but small gaps vs. baselines across benchmarks)
- Remove the graph view → relational-question accuracy drops significantly
- Remove the temporal hierarchy → temporal-question accuracy drops
- Remove dual-view coordination (single view only) → overall performance drops
- Remove deterministic calibration → answer trustworthiness drops
The graph records "observed facts"—who was mentioned where—not LLM-inferred relations.
When you ask "how is the project I discussed with Xiao Wang last week," Zero-Mem doesn't ask an LLM to summarize everything about Xiao Wang. Instead it:
1. Extracts the entity "Xiao Wang" from the query 2. Finds the "Xiao Wang" node in the graph 3. Spreads activation along edges—all dialogue segments containing "Xiao Wang" 4. Follows adjacency edges to surrounding context
This uses Personalized PageRank, not LLM generation.
View 2: Temporal Hierarchy (local view)
Graphs find relations, not time order. Zero-Mem also organizes dialogue hierarchically:
Asking "what did we discuss around 3pm yesterday" prioritizes this view, narrowing from Episode to Window to Turn.
Dual-view coordination
Key design: different questions need different views.
A lightweight query analyzer (not an LLM) classifies the question and weights the two views. Retrieved results are fused, then pass through an "evidence closure" step that adds relational links and local context.
Deterministic calibration: the last line of defense
Before evidence reaches the answering LLM, two deterministic calibration steps run:
1. Evidence calibration: discard conflicting evidence (keep the latest for facts that changed), filter content not supporting the question 2. Answer calibration: after the LLM answers, check the answer is evidence-supported, type-matched, and correctly formatted
Both are deterministic rules—no LLM calls.
Key Numbers: Zero Tokens + 57.6% Latency Reduction
Tested on multiple long-memory and long-context QA benchmarks:
57.6% is worth unpacking: Zero-Mem saves not just token costs but time. For fast-response agent scenarios, halving memory-operation latency is a qualitative UX improvement.
Ablations: Both Views Are Necessary
Deeper Significance
Zero-Mem touches a fundamental question in agent design: what should the LLM do?
The current trend is "LLM for everything"—memory management, tool selection, result verification. Every step becomes an LLM call; costs and latency stack up.
Zero-Mem proposes the opposite:
> The LLM should only do what it's best at: understanding questions and generating answers. All structured, rule-encodable, algorithmizable operations should use dedicated tools.
This echoes Euclid-MCP's "let the LLM be the poet, let Prolog be the accountant," and Rebucca's "small model triage + large model verification." Division of labor beats unification—not a new discovery, but Zero-Mem pushes it to the extreme: LLM calls for memory operations go straight to zero.
Another notable point is provenance. Every memory unit keeps raw text, source ID, and timestamp, so any answer can be traced to original dialogue. As AI systems demand more auditability, this design matters—you can't explain one LLM's answer with another LLM's summary, because stacked hallucinations can't be traced.
Limitations and Outlook
1. NER dependence: spaCy's quality determines the graph view. For Chinese and multilingual scenarios, NER may be weaker than English. 2. Complex reasoning needs more: some memory operations genuinely need inference (e.g., inferring preference changes from three months of behavior); pure retrieval may not suffice. 3. Limited benchmarks: the paper tests QA benchmarks; real agent scenarios (multi-turn tool calls, multimodal input) remain unvalidated.
But Zero-Mem's core contribution isn't beating baselines—it's proving feasibility: structured memory operations can work with zero LLM generation. That proof opens the door for more complex structured operations on top of "zero generation."
Closing
Zero-Mem recalls an old engineering principle: the best code is no code. Likewise, the best LLM call is no call. When an operation doesn't need an LLM, you save money and gain determinism, auditability, and speed.
As agent systems grow more complex and LLM calls denser, Zero-Mem reminds us: not every problem needs generation. Sometimes you just need a good catalog system.
---
Paper: https://arxiv.org/abs/2607.29377 Code: https://github.com/TheMoon0815/Zero-mem