Zero-Mem: Zero-Token Memory Operations for AI Agents

Forum topic · ✨步子哥 · 2026-08-04

Summary

Zero-Mem is a new memory architecture for AI agents that eliminates LLM calls during memory operations. Instead of using generative summarization, extraction, and retrieval with large language models, Zero-Mem keeps every original interaction trace and builds two complementary indexes: an entity-context graph built with a lightweight NER tool (such as spaCy) plus Personalized PageRank for relationship retrieval, and a hierarchical turn/window/episode/local timeline for temporal queries. A non-LLM query analyzer weighs the two views, and deterministic calibration filters conflicts and unsupported evidence. Reported results include zero LLM calls and zero LLM tokens for memory operations, a 57.6% reduction in memory-operation latency versus the fastest baseline, and competitive accuracy on long-memory QA benchmarks. The work argues that LLMs should focus only on question understanding and answer generation, while structured tasks should rely on algorithms and rules for lower cost, lower latency, determinism, and full provenance.

Key points

Problem — the "generation tax" of current agent memory. Mainstream memory stacks (Zep, Mem0, A-Mem, MemoryOS, GAM) rely on the LLM to summarize, extract, update, and retrieve memories. Each step burns input + output tokens, and every compression layer hides original evidence ("generation is loss"), breaking the provenance chain between an answer and its source.
Zero-Mem's core idea: no generation, only structured selection. Every original interaction trace is preserved verbatim. Two complementary views are built on top of the raw traces, and memory operations become deterministic algorithms instead of LLM calls.
View 1: entity-context graph (relational view). A lightweight NER model (e.g., spaCy, not an LLM) extracts named entities from each conversation. Edges connect entity nodes to context nodes, and adjacent context nodes to each other, recording observed facts, not inferred relationships. Retrieval uses Personalized PageRank: activate the entity node, propagate along edges to find relevant contexts, then expand to adjacent turns.
View 2: temporal hierarchy (local view). Conversations are organized as Turn → Window → Episode → Local neighborhood. Temporal queries drill down from episode to window to turn to find the time-bounded slice.
Dual-view coordination. A lightweight, non-LLM query classifier decides whether the question is relational, temporal, or mixed, and assigns weights to each view. Results are fused and passed through an "evidence closure" step that adds bridging relations and local neighborhood context.
Deterministic calibration as the last line of defense. Evidence calibration drops conflicting facts (keeping the latest) and filters content that does not support the question. Answer calibration checks that the LLM's final answer is evidence-backed, type-correct, and format-correct. Both steps are rule-based and make no LLM call.

Reported results

LLM calls during memory operations: 0 (vs. dozens in baselines)
LLM tokens consumed by memory operations: 0 (encoder inference is billed separately)
Memory-operation latency: −57.6% vs. the fastest baseline under the same final QA reader and equal context budget
Answer quality: competitive, not SOTA, but close to strong baselines on long-memory QA benchmarks

Ablation findings

Removing the graph view → large drop on relational questions
Removing the temporal hierarchy → large drop on temporal questions
Removing dual-view coordination → overall accuracy drops
Removing deterministic calibration → answer trustworthiness drops

The two views are complementary, not redundant.

Why it matters

Rethinking the LLM's role. Zero-Mem argues that LLMs should only handle what they are best at: understanding the question and generating the answer. Everything that is structural, rule-based, or algorithmic should be delegated to dedicated tools. This is the same "division of labor" philosophy behind Euclid-MCP ("LLM as poet, Prolog as accountant") and Rebucca's "small-model pre-filter + large-model verify" pattern, taken to its logical endpoint: zero LLM calls for memory ops.
Provenance and auditability. Because raw traces are preserved with source and timestamp, every answer is traceable to original conversations, critical for systems that must be auditable.
Cost, latency, determinism. Dropping LLM calls removes stochastic generation steps, yielding reproducible memory operations and lower end-to-end latency.

Limitations and outlook

1. NER quality (here spaCy) caps the graph view; Chinese and multilingual coverage may be weaker than English. 2. Pure retrieval cannot replace reasoning in every memory task, such as inferring preference drift across months. 3. Evaluation is on QA benchmarks; real agent workloads with multi-turn tool use and multimodal input remain to be tested.

The contribution is not "beating baselines by X%" but a feasibility proof: structured memory operations can be done entirely without generation, opening a design space where richer structured operations can be stacked on top of a generation-free foundation.

Resources

Paper: https://arxiv.org/abs/2607.29377
Code: https://github.com/TheMoon0815/Zero-mem

FAQ

Who is this for? Practitioners, researchers, and students interested in AI agents, LLM systems, and memory architectures.

What are the core takeaways?

The "generation tax" of today's LLM-based memory systems
Zero-Mem's insight: do not generate, only select
View 1: entity-context graph for relational retrieval

Is the code open source? Yes, see the link above.