Mapping 297 Memory Architectures—and a Mostly Empty Top Floor

Source

Survey: "Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory" (arXiv 2610.19366, posted 2026-10-14). 80 authors including Minghao Guo, Qingyue Jiao, and Xiujin Liu, with corresponding authors from Rutgers, Notre Dame, and Fudan University. Companion repository: Seeing-Maintaining-Learning/multimodal-agent-memory-evolution (created 2026-09-29, MIT license, taxonomy only—no code released; numbers still being updated).

A lifetime-based view of memory

Instead of cataloging components, the survey looks at a memory's entire life: how experience becomes memory, how memory is maintained, and how maintenance policies themselves are learned. It gives a uniform definition—memory is any experience-derived state that persists beyond the step that produced it and influences subsequent reasoning and action. Under this definition, all of the following are memory:

• KV cache: zero distillation, full retention—the crudest form
• External text/structured note stores
• State compressed into parameters

This ends the habit of three research camps talking past each other.

The map: two axes

Axis 1 — what is stored, and how:

• Native source-modality representations (store embeddings as-is): high fidelity, expensive storage
• Text and structured records (store distilled text): editable and reusable, but distillation loses detail
• Implicit parameterized representations (store in weights/state): compact but hard to audit
• Hybrid representations: combinations of the above

Axis 2 — a three-tier ladder for maintenance:

• T1 static: stored and never touched again
• T2 dynamic: content updated by a fixed policy
• T3 self-evolving: the update policy itself evolves with experience

Where recent systems sit

Placing recently followed systems on the map shows a tidy pattern:

• KV-cache-to-memory-pool systems: native representation + T1
• Four-tier memory stores like Hindsight: text representation + T2 (content is refined, but the refinement procedure is hardcoded)
• MemT-1.0: parameterized representation + T2 (content updates every round; the policy is gradient-trained into weights and then equally fixed)

Everything lands in T1 or T2. And the survey's first cross-cutting law confirms it: T3 is scarce—systems that keep learning "how to manage memory" after deployment can be counted on one hand across the whole field.

The other two cross-cutting laws: dynamic updating (T2) is already ubiquitous, and hybrid representations are widely adopted.

Six future directions

Of the six directions, four say two sides of the same thing—memory must keep evolving and keep being audited:

1. Self-improving memory management (formalizing T3) 2. Provenance-traceable memory: every memory must point back to its original evidence—an honesty mechanism for memory (what was forgotten, what was kept, on what grounds—all checkable) 3. Governed forgetting: forgetting is a capability, not a failure—but it must be accountable 4. Bounded lifelong memory: memory cannot expand without limit

The empty top floor

The vacant T3 tier connects to an old problem in the self-improvement line: weights can be trained, code can be searched, policies can evolve—but "memory-management policy" as an object of improvement has barely been touched. Whoever moves into the top floor first must answer the most expensive old question: who verifies whether the improvement was correct?

The count of 297 is the current version's number; the three-tier map will be updated, and so will the top floor's tenant list. Among the 80 authors, someone may already be moving up.

*Source notes: figures (297 architectures, 82 benchmarks, T1/T2/T3 definitions, three cross-cutting laws, six directions) verified against the paper's introduction and body. The companion GitHub repo is a taxonomy repository (no code); material claiming "100+ authors" was corrected to 80.*