Zero-Mem: Eliminating LLM Calls from AI Agent Memory Operations

Forum topic · ✨步子哥 · 2026-08-03

Summary

Zero-Mem is a novel AI agent memory system that performs all memory operations—storage, retrieval, and updating—without invoking large language models, achieving zero-token memory costs and a 57.6% latency reduction compared to leading baselines. Rather than relying on LLMs to summarize, extract, or compress conversations (incurring recurring "generation tax" and hallucination risk), Zero-Mem preserves raw interaction traces and overlays two complementary structured views: a lightweight entity-context graph built via non-LLM NER (such as spaCy) using Personalized PageRank for relational queries, and a temporal hierarchy (Turn → Window → Episode → Local) for time-based queries. A deterministic query analyzer routes weight between the two views, and deterministic calibration filters evidence and validates answers without LLM calls. Evaluations on long-memory and long-context QA benchmarks show competitive accuracy with the strongest baselines while reducing LLM calls during memory operations from dozens to zero. Ablation studies confirm both views are essential. The work demonstrates that structured, algorithm-based methods can replace generative methods for memory management.

Overview

Zero-Mem is a novel AI agent memory system that performs all memory operations—storage, retrieval, and updating—without invoking large language models. It achieves zero-token memory costs and a 57.6% latency reduction compared to leading baselines such as Zep, Mem0, A-Mem, MemoryOS, and GAM, while keeping competitive QA accuracy.

Paper: https://arxiv.org/abs/2607.29377
Code: https://github.com/TheMoon0815/Zero-mem

The Problem: "Generation Tax" in Existing Memory Systems

Current agent memory systems rely on LLMs to manage memory through four recurring operations:

1. Summarization — compressing long dialogues into bullet points 2. Extraction — pulling entities, relations, and events from conversations 3. Update — merging or modifying old memories when new information arrives 4. Retrieval — using LLMs to judge which memories are relevant to a query

Each call consumes input + output tokens. Over hundreds of dialogue turns, this "generation tax" compounds. More critically, every generative step introduces information loss and hallucination, breaking the provenance chain from raw evidence to final answer.

Core Insight: Selection, Not Generation

> Memory operations do not require generation—only structured selection.

Like a library that shelves books intact and uses only a catalog system, Zero-Mem keeps every raw interaction trace untouched and overlays two complementary structured views.

Two Views Over Raw Traces

View 1: Entity-Context Graph (relational view)

A lightweight NER model (e.g., spaCy) extracts entities (people, places, organizations) from each dialogue segment.
A graph connects entity nodes to context nodes, and neighboring context nodes to each other.
Retrieval uses Personalized PageRank (a graph algorithm), not LLM generation.
Records only "observed facts"—who was mentioned where—not inferred relations.

Example query: *"How is the project with Xiao Wang progressing?"*

Extract entity "Xiao Wang" from the query.
Locate the entity node, propagate activation along edges.
Collect all context fragments containing the entity, then expand to adjacent windows.

View 2: Temporal Hierarchy (local view)

Conversation traces are organized into nested layers:

Turn — a single Q&A exchange
Window — a few adjacent turns
Episode — several windows forming a complete event
Local — the immediate context of a candidate evidence passage

Example query: *"What did we discuss around 3 PM yesterday?"* The system descends from Episode → Window → Turn to localize the relevant time range.

View Coordination

A lightweight (non-LLM) query analyzer classifies the question type and assigns weights to each view:

| Query type | Example | Dominant view | |---|---|---| | Relational | "Xiao Wang's project status" | Graph view | | Temporal | "What did we discuss yesterday afternoon?" | Hierarchy view | | Hybrid | "What happened after Xiao Wang postponed the project?" | Both views |

Results from both views are fused, then passed to evidence closure—a step that augments relational links and local context.

Deterministic Calibration: The Last Line of Defense

Before the final answer LLM ever sees the evidence, two deterministic (non-LLM) checks run:

1. Evidence calibration — discard conflicting evidence (keep the most recent value for the same fact); drop passages that do not support the query. 2. Answer calibration — verify the final answer is grounded in evidence, type-matches the question, and follows formatting rules.

Key Results

Evaluated on long-memory and long-context QA benchmarks:

LLM calls for memory operations: 0 (baselines: dozens)
LLM token consumption for memory operations: 0 (encoder costs excluded)
57.6% memory-operation latency reduction vs. the fastest baseline (with identical QA reader and equal context budget)
Competitive QA accuracy—not the strongest, but close to baselines on multiple benchmarks

Ablation Findings

| Component removed | Effect | |---|---| | Graph view | Relational QA accuracy drops sharply | | Temporal hierarchy | Temporal QA accuracy drops | | Dual-view coordination | Overall accuracy degrades | | Deterministic calibration | Answer trustworthiness decreases |

The result is not "less work, therefore faster"—it is "structured methods replace generative methods while preserving effectiveness."

Deeper Significance

Zero-Mem poses a foundational design question: what should LLMs actually do?

Today's trajectory is "everything is an LLM"—memory management, tool selection, result verification—all become LLM calls, compounding cost and latency. Zero-Mem argues the opposite:

> LLMs should do what they do best: understand questions and generate answers. Everything that is structured, rule-based, or algorithmic should be handled by dedicated tools.

The approach aligns with Euclid-MCP's "let the LLM be the poet, let Prolog be the accountant" philosophy and with Rebucca's "small-model filter + large-model verify" pattern. Division of labor beats uniformity.

A standout property is provenance: every memory unit retains original text, source identifier, and timestamp, making any final answer traceable back to raw dialogue. As AI systems face increasing audit demands, this matters—you cannot debug one LLM-generated summary by citing another LLM-generated summary.

Limitations

1. NER dependency — graph quality depends on the external NER model; spaCy's performance on Chinese and multilingual data trails its English results. 2. Limited reasoning — pure retrieval may not handle queries that require genuine inference (e.g., inferring preference shifts from three months of behavior). 3. Benchmark scope — evaluations cover QA benchmarks; real agent scenarios (multi-turn tool use, multimodal input) remain to be validated.

Still, the core contribution is not beating baselines on accuracy—it is proving feasibility: structured memory operations can be entirely LLM-free.

Closing Thought

The engineering adage *"the best code is no code"* has an LLM-era counterpart: the best LLM call is no call. When an operation does not require generation, you gain not only cost savings but also determinism, auditability, and speed. Zero-Mem is a timely reminder that not every problem needs a generative solution—sometimes all you need is a good catalog.