Over the past two years, equipping AI agents with memory has become a hot research direction: layered storage, knowledge graphs, memory lifecycle management — architectures of every kind have taken the stage. A team at the University of Illinois Urbana-Champaign asked a basic question that had been skipped: how much better is adding memory than doing nothing at all and simply retrieving from raw records?
They benchmarked 11 representative memory systems on three long-horizon interaction benchmarks and three model backbones, against the plainest possible baseline: chunk the raw conversations, index them, retrieve by similarity — textbook RAG with no memory processing whatsoever.
The result: in 99 comparisons, 90 were lost to this naive baseline. Not one of the eleven systems won across the board.
This Is Not a Staged Hit Piece on RAG
The experimental fairness design is the most instructive part of the paper.
Memory systems differ wildly in how much content they retrieve: some return a few summaries, others dump a large blob. Past comparisons used unequal retrieval budgets, which skewed conclusions. This paper set a uniform scale: each system's retrieved content was measured as a proportion of tokens exposed to the model relative to its retrievable corpus, and systems were tuned to at least the RAG level. Memory systems were not shortchanged — they got a generous budget. They still lost.
The three benchmarks all test long-horizon interaction memory: LongMemEval-s for information access in multi-turn long-range dialogue, PersonaMem-v2 for cross-session user profiles and preferences, and MINT-Eval, from a 2026 work. The three backbones range from 4B to 27B to a lower tier of a closed-source flagship model.
The Casualty List
Of 99 matchups, only 9 wins were scattered across 6 systems, with the best single system winning 3, at a maximum advantage of +3.9 percentage points. Everything else was negative.
Some losses were decisive. A-Mem, SimpleMem, HippoRAG2, MAGMA, and OpenVingo — five systems — scored below RAG in all nine combinations. OpenViking trailed by 40 percentage points on long-horizon dialogue: the more elaborate the layered database architecture, the harder the fall.
Memory-R2, a training-based approach that uses reinforcement learning specifically to train memory updating, still went negative in 8 of 9 matchups. Even trained systems didn't escape.
Don't Throw Out Memory Yet
It's easy to conclude "memory is useless." The paper's controls close off that conclusion too.
Stuffing all raw records into the context (Full Context) performed *worse* than retrieval on the user preference benchmark — small models get overloaded and fail at questions they should answer. Giving nothing (No Context) still scores 9 to 29 points, meaning a substantial share of these benchmark questions can be answered by general capability alone.
Gold Context (human-selected perfect context) reaches 73 to 93 points. That number shows the information exists and good answers exist — what breaks is the processing in the middle.
So the question is not "whether to have memory," but whether the memory-processing step has a net positive effect.
Benefits and Taxes
The paper's core framework decomposes this step into two ledgers: benefits and taxes.
Any memory processing deviates from the raw history: distilling conversations into atomic facts, linking sessions into graphs, distilling preferences into profiles. The deviation is intentional, designed for faster lookup and better answers — that's the benefit. But deviation has another side: some evidence is lost during compression, some is rewritten so it no longer matches the source, some is buried deep in a graph where retrieval can't reach it — that's the tax.
The paper traces three sources of tax:
- Fidelity: compression and rewriting during construction lose evidence, and what's lost can't be recovered later. Sampling and manually annotating the retention of atomic facts in memory units confirmed that information loss genuinely exists in most systems.
- Reachability: evidence that survives but is buried is useless. Measured by first-round retrieval hit rate, memory systems generally hit less than RAG — evidence buried by their own structure.
- Robustness: rebuilding memory with one backbone and answering with another causes clear score swings; both the construction side and the answering side amplify the tax.
- Make deviation recoverable. Since processing can lose or hide evidence, leave a light path back to the source: run memory retrieval as usual while also retrieving from the raw records, merge both and re-rank before answering. The implementation is just one extra retrieval branch in the pipeline — several systems that adopted it saw consistent score gains.
- Recursive retrieval. When first-round retrieval results are too thin, automatically dig another layer. Evidence that was unreachable becomes reachable with one more round.
One evaluation-side finding: the easier the benchmark, the more visible the tax. When a benchmark still has plenty of headroom, processing-induced losses are amplified in scores — the difficulty of the questions you pick determines whether a memory system looks like a win or a loss.
The framework extends beyond conversational memory. Applied to experiential memory (agents accumulating task experience): distilling trajectories into workflows *beats* retrieval on web tasks, while several methods fall behind on tool-calling tasks — the same mechanism, in a different domain, turns benefit into tax. World-model memory shows the same pattern: keeping raw frames versus compressing into latent vectors, the same deviation, the same double edge.
How to Fix It
The paper's design suggestions deliberately avoid "invent another new architecture," and both are lightweight:
The paper was posted October 25, with code and all results promised open source; as of writing, not yet released. Until then, the 11×3×3 scorecard can only be viewed on the project page.
One side note: two names on this team are worth remembering — Jiawei Han and ChengXiang Zhai, two decades in data mining and two decades in information retrieval, respectively. Treating retrieval as the baseline seems to be instinct for them.
---
*Paper: Memory Tax: Dissecting the Net Value of Agent Memory (arXiv:2610.34073, submitted 2026-10-25). Project page: hexuan21.github.io/MemoryTax. All experimental figures in this article come from the paper and the project page leaderboard.*