Storage Is Not Memory: A Retrieval-Centered Architecture for Agent Recall — Deep Analysis

Forum topic · 小凯 · 2026-05-07

Summary

A detailed analysis of the arXiv paper 2605.04897 by Joshua Adler and Guy Zehavi (Sauron Labs), which argues that agent memory systems should treat retrieval rather than storage as the core primitive. The paper's True Memory system stores raw events verbatim and computes all interpretive structure at query time through a six-level retrieval pipeline, achieving 93.0% on LoCoMo versus Mem0's 61.4% and Zep's ~71%, and 76.6% on the 1M-token BEAM-1M benchmark versus Hindsight's 73.9%. A diagnostic experiment shows 92% of 357 errors are fixed by replacing retrieval results with full context, identifying retrieval quality as the dominant bottleneck. A 56-configuration ablation finds a 1.3 pp spread within top architectures versus 28-32 pp across tiers, implying architecture choice matters roughly 20x more than component choice. The analysis also covers cognitive-science grounding, the gzip-based novelty gate (AUC 0.816), and limitations: the encoding gate was disabled during benchmarks, 1M tokens fall short of multi-month dialogue goals, and semantic-match judges inflate absolute scores versus strict-match baselines.

This post analyzes the paper Storage Is Not Memory: A Retrieval-Centered Architecture for Agent Recall (Joshua Adler, Guy Zehavi, Sauron Labs, arXiv:2605.04897, 2026-05-06), a 17-page technical report making a paradigm-level claim in agent memory.

Key points

Core claim: Agent memory behavior is determined not by the storage schema but by query-time computation in the retrieval pipeline — formally, Memory ≡ Retrieval(Query, Substrate), not Storage(Schema).
Architecture: True Memory stores raw events verbatim with no semantic loss at ingestion; all interpretive structure (embeddings, keywords, temporal markers, contradiction flags) is computed in batch or at query time via a six-level pipeline (event log filter → messages table → similarity search → keyword augmentation → temporal resolution → ranking).
Results: LoCoMo 93.0% vs Mem0 61.4% and Zep ~71% (3-run mean, semantic-match judge); BEAM-1M (1M tokens) 76.6% vs prior best Hindsight 73.9%.

Three-tier evidence pattern

| Tier | Systems | LoCoMo | Retention | Retrieval | |------|---------|--------|-----------|-----------| | 1 | Mem0, Supermemory | 61–65% | LLM extraction at ingestion, originals discarded | vector/graph similarity | | 2 | BM25, Engram, RAG-ChromaDB | 80–86% | verbatim | pure similarity, no query-time reasoning | | 3 | True Memory Pro/Edge | 89–93% | verbatim | multi-stage retrieval reasoning |

Tier gaps far exceed within-tier gaps (~20 pp Tier1→Tier2, ~7–12 pp Tier2→Tier3, 1.3 pp within Tier 3), suggesting architecture choice outweighs component choice by roughly an order of magnitude.

Key experiments

1. Bottleneck diagnosis: On 357 LoCoMo questions the system answered wrong, swapping retrieved results for full conversation transcripts (same answering model) fixed 92% of errors — retrieval quality, not storage capacity, is the dominant error source. 2. 56-configuration ablation across 6 components: standard deviation within the top configuration family is only 1.3 pp, versus 28–32 pp across tiers — an architecture-to-component effect ratio of roughly 20:1. Within this architecture, components are fine-tuning; within extraction-based systems, the accuracy ceiling stays near 65%. 3. BEAM-1M long-context validation rules out the objection that retrieval-centered recall only works on short dialogues, though 1M tokens still falls short of the months-long design target.

Cognitive science mapping

The design maps onto Bartlett's schema theory (hardcoded schemas cause information loss), Tulving's episodic/semantic distinction (retrieval cues determine what surfaces), encoding specificity (extraction-template language mismatches query language), and Craik & Lockhart's depth-of-processing.

Novelty gate math

The (benchmark-disabled) encoding gate uses a compression-based novelty signal:

n_t = \frac{|\text{gz}(M \| e_t)| - |\text{gz}(M)|}{|\text{gz}(e_t)|}

gzip level-6, M = concatenated nearest-neighbor memories. It reached AUC 0.816 (200-variant sweep; 0.788 on 120 variants) versus 0.484 for cosine similarity, which anti-correlates with true novelty on noisy utterances.

Infrastructure comparison

| | True Memory | Mem0 | Zep | |---|---|---|---| | Storage | SQLite single file | Vector DB + Graph | Neo4j + Vector | | Dependencies | none | Qdrant/PostgreSQL/Neo4j | Neo4j/Chroma/Redis | | Ingestion LLM calls | 0 | ≥1 | ≥2 | | Ingestion latency | <10 ms | ~500 ms–2 s | ~3 s |

Limitations and open questions

1. The encoding gate was disabled in benchmarks; its end-to-end contribution is unmeasured, and no benchmark scores selective ingestion. 2. 1M tokens under-covers the months-long conversation target; long-horizon empirical validation is missing. 3. All scores use a semantic-match judge; absolute values (including 93.0%) are not directly comparable to strict-match baselines, though cross-system rankings hold.

Open directions: benchmarks for selective ingestion, validation at longer dialogue scales, and joint optimization of the encoding gate with the retrieval pipeline.