> Paper: MEMO: Memory as a Model > arXiv: 2605.15156 > Team: NUS · MIT CSAIL · A*STAR · SMART > Core claim: move long-term memory from a "vector-store plug-in" to a trainable, swappable memory model decoupled from the base LLM.
Key points
- Problem with existing approaches: Non-parametric methods (RAG, ICL) are sensitive to retrieval noise, weak at cross-document reasoning, and bounded by context length. Parametric methods (continued pretraining, SFT) are expensive, cause catastrophic forgetting, and cannot touch closed models. Implicit memory (soft tokens: AutoCompressor, Gist, ICAE) suffers from representation coupling — memories are bound to the model that produced them and cannot transfer.
- Architecture: A small MEMORY model (1.5B–14B, e.g., Qwen2.5-14B-Instruct) internalizes a corpus into its parameters via supervised fine-tuning, while a frozen black-box EXECUTIVE LLM (32B, 70B, or even Claude/Gemini — no weight or logit access needed) drives reasoning. Memory and inference are fully decoupled: either side can be swapped without retraining the other.
- Five-step data pipeline: (1) direct and indirect fact extraction; (2) merging QA pairs sharing context into multi-fact questions; (3) verification/rewriting for self-containedness; (4) entity surfacing ("indirect description → entity" QA) to counter the reversal curse; (5) cross-document synthesis capturing convergent clues and parallel attributes. No document identifiers or watermarks are embedded, preventing shortcuts.
- Training: next-token prediction on answer tokens only, conditioned on questions without source documents (3 epochs, Fused AdamW, LR 2×10⁻⁵). ~90–180 GPU-hours (H200) per corpus.
- Three-stage inference protocol: (1) grounding — EXECUTIVE decomposes the query into atomic sub-questions answered independently by MEMORY; (2) entity identification — iterative follow-ups narrow candidates; (3) answer seeking and synthesis — EXECUTIVE queries supporting facts and generates the final answer itself. MEMORY responses are compact natural-language snippets, giving constant-time inference regardless of corpus size.
- Data generation cost: ~240 GPU-hours for QA synthesis + ~180 for training ≈ 420 GPU-hours per corpus — costlier than RAG indexing, cheaper than full SFT.
- Generalization: questions outside the GENERATOR's synthesized distribution may underperform (a shared weakness of parametric approaches).
- RAG is still improving: HippoRAG2 retains an edge on pure retrieval precision; MEMO's advantage is cross-document reasoning and noisy-retrieval settings.
- API costs: black-box EXECUTIVEs still require API access, which can be expensive at high call volumes.
Benchmark results
BrowseComp-Plus: MEMO 54.22% (Qwen2.5-32B EXECUTIVE) / 66.67% (Gemini-3-Flash), vs. HippoRAG2 56.11% / 66.33% and Perfect Retrieval ceiling 79.67% / 88.33%.
NarrativeQA: MEMO 26.85% / 53.58% vs. HippoRAG2 21.39% / 23.21%; ceiling 51.42% / 60.41%.
MuSiQue: MEMO 48.30% / 60.20% vs. HippoRAG2 42.17% / 57.00%; ceiling 62.83% / 73.00%.
Notable findings:
1. Cross-model transfer: swapping EXECUTIVE from Qwen2.5-32B to Gemini-3-Flash with the *same* MEMORY model gains +12.45%, +26.73%, +11.90% across the three datasets. 2. Noise robustness: adding retrieval noise drops HippoRAG2 by 6.22% on BrowseComp-Plus; MEMO changes by +0.55% (within error) since it does no retrieval. 3. Small models work: a 1.5B MEMORY model reaches 21.16% on NarrativeQA, close to HippoRAG2's 21.39%.
Continual integration via model merging
Each new corpus trains a separate MEMORY model initialized from the same base; task vectors τ_i = φ_i − φ₀ are merged (linear weighting, TIES). TIES saves 33% compute at K=2 corpora and 5.5× at K=10, with a measurable but acceptable accuracy drop vs. full retraining — enabling streaming knowledge updates.
Method comparison
| Property | RAG | SFT | Implicit memory | MEMO | |---|---|---|---|---| | Frozen base LLM | ✓ | ✗ | ✓ | ✓ | | No retrieval index | ✗ | ✓ | ✓ | ✓ | | Black-box compatible | ✓ | ✗ | ✗ | ✓ | | No catastrophic forgetting | ✓ | ✗ | ✓ | ✓ | | Constant-size memory | ✗ | ✓ | ✗ | ✓ | | Cross-model transferable | ✓ | ✗ | ✗ | ✓ |
MEMO is the only method satisfying all six.
Limitations
Why it matters
MEMO reframes memory as an independently trainable, deployable, and swappable module — analogizing the move from coprocessor to standalone compute unit. One MEMORY model can serve many EXECUTIVE models, letting enterprises internalize a knowledge base once and call it from any LLM. And if a 1.5B model can memorize a full corpus, personal knowledge bases could become tiny models attached to Claude or GPT — no vector DB, no retrieval pipeline, no chunk-boundary semantic breaks.
Conclusion
MEMO is not a RAG replacement but a different route: knowledge is invoked at inference time rather than pulled at retrieval time. It has structural advantages for cross-document synthesis, retrieval-noise robustness, and multi-EXECUTIVE support; RAG remains more practical for pure retrieval precision, cost-sensitive, or out-of-distribution query scenarios. Still, the "memory as a model" paradigm is a clear milestone in knowledge-injection architecture.
> References > - arXiv: 2605.15156 — Quek et al., "MEMO: Memory as a Model", May 2026 > - Marktechpost coverage: https://www.marktechpost.com/2026/05/26/memo-a-modular-framework/ > - NeurIPS 2023 LongMem paper (related work on decoupled memory)