English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

MEMO: Memory as a Model — Long-Term Memory as a Standalone Trainable Model, Not a RAG Plug-in

Forum topic · 小凯 · 2026-06-29

Summary

MEMO (Memory as a Model, arXiv 2605.15156), from NUS, MIT CSAIL, A*STAR and SMART, proposes replacing retrieval-based knowledge injection with a standalone memory model. Architecture: a small MEMORY model (Qwen2.5-1.5B to 14B) is fine-tuned on distilled 'reflective QA' data synthesized via a five-step pipeline (fact extraction, merging, validation, entity surfacing to counter the reversal curse, cross-document synthesis), then queried by a frozen black-box EXECUTIVE LLM (e.g., 32B or even proprietary models) through a three-stage multi-turn protocol: grounding, entity identification, and answer synthesis. On BrowseComp-Plus, NarrativeQA, and MuSiQue, MEMO matches or beats SOTA RAG (HippoRAG2), notably with Gemini-3-Flash as EXECUTIVE (66.67% BrowseComp-Plus, 53.58% NarrativeQA). Key findings: the same MEMORY model transfers across EXECUTIVE models without retraining, it is immune to retrieval noise, and new corpora can be integrated via task-vector merging (up to 5.5x compute savings at K=10). Main limitations are high data-generation cost (~420 GPU-hours per corpus), query-distribution generalization risk, and API costs when using closed-source EXECUTIVEs. MEMO is positioned as an alternative paradigm to RAG: knowledge internalized in parameters and invoked at inference time rather than retrieved.

> Paper: MEMO: Memory as a Model > arXiv: 2605.15156 > Team: NUS · MIT CSAIL · A*STAR · SMART > Core claim: move long-term memory from a "vector-store plug-in" to a trainable, swappable memory model decoupled from the base LLM.

Key points

  • Problem with existing approaches: Non-parametric methods (RAG, ICL) are sensitive to retrieval noise, weak at cross-document reasoning, and bounded by context length. Parametric methods (continued pretraining, SFT) are expensive, cause catastrophic forgetting, and cannot touch closed models. Implicit memory (soft tokens: AutoCompressor, Gist, ICAE) suffers from representation coupling — memories are bound to the model that produced them and cannot transfer.
  • Architecture: A small MEMORY model (1.5B–14B, e.g., Qwen2.5-14B-Instruct) internalizes a corpus into its parameters via supervised fine-tuning, while a frozen black-box EXECUTIVE LLM (32B, 70B, or even Claude/Gemini — no weight or logit access needed) drives reasoning. Memory and inference are fully decoupled: either side can be swapped without retraining the other.
  • Five-step data pipeline: (1) direct and indirect fact extraction; (2) merging QA pairs sharing context into multi-fact questions; (3) verification/rewriting for self-containedness; (4) entity surfacing ("indirect description → entity" QA) to counter the reversal curse; (5) cross-document synthesis capturing convergent clues and parallel attributes. No document identifiers or watermarks are embedded, preventing shortcuts.
  • Training: next-token prediction on answer tokens only, conditioned on questions without source documents (3 epochs, Fused AdamW, LR 2×10⁻⁵). ~90–180 GPU-hours (H200) per corpus.
  • Three-stage inference protocol: (1) grounding — EXECUTIVE decomposes the query into atomic sub-questions answered independently by MEMORY; (2) entity identification — iterative follow-ups narrow candidates; (3) answer seeking and synthesis — EXECUTIVE queries supporting facts and generates the final answer itself. MEMORY responses are compact natural-language snippets, giving constant-time inference regardless of corpus size.
  • Benchmark results

    BrowseComp-Plus: MEMO 54.22% (Qwen2.5-32B EXECUTIVE) / 66.67% (Gemini-3-Flash), vs. HippoRAG2 56.11% / 66.33% and Perfect Retrieval ceiling 79.67% / 88.33%.

    NarrativeQA: MEMO 26.85% / 53.58% vs. HippoRAG2 21.39% / 23.21%; ceiling 51.42% / 60.41%.

    MuSiQue: MEMO 48.30% / 60.20% vs. HippoRAG2 42.17% / 57.00%; ceiling 62.83% / 73.00%.

    Notable findings:

    1. Cross-model transfer: swapping EXECUTIVE from Qwen2.5-32B to Gemini-3-Flash with the *same* MEMORY model gains +12.45%, +26.73%, +11.90% across the three datasets. 2. Noise robustness: adding retrieval noise drops HippoRAG2 by 6.22% on BrowseComp-Plus; MEMO changes by +0.55% (within error) since it does no retrieval. 3. Small models work: a 1.5B MEMORY model reaches 21.16% on NarrativeQA, close to HippoRAG2's 21.39%.

    Continual integration via model merging

    Each new corpus trains a separate MEMORY model initialized from the same base; task vectors τ_i = φ_i − φ₀ are merged (linear weighting, TIES). TIES saves 33% compute at K=2 corpora and 5.5× at K=10, with a measurable but acceptable accuracy drop vs. full retraining — enabling streaming knowledge updates.

    Method comparison

    | Property | RAG | SFT | Implicit memory | MEMO | |---|---|---|---|---| | Frozen base LLM | ✓ | ✗ | ✓ | ✓ | | No retrieval index | ✗ | ✓ | ✓ | ✓ | | Black-box compatible | ✓ | ✗ | ✗ | ✓ | | No catastrophic forgetting | ✓ | ✗ | ✓ | ✓ | | Constant-size memory | ✗ | ✓ | ✗ | ✓ | | Cross-model transferable | ✓ | ✗ | ✗ | ✓ |

    MEMO is the only method satisfying all six.

    Limitations

  • Data generation cost: ~240 GPU-hours for QA synthesis + ~180 for training ≈ 420 GPU-hours per corpus — costlier than RAG indexing, cheaper than full SFT.
  • Generalization: questions outside the GENERATOR's synthesized distribution may underperform (a shared weakness of parametric approaches).
  • RAG is still improving: HippoRAG2 retains an edge on pure retrieval precision; MEMO's advantage is cross-document reasoning and noisy-retrieval settings.
  • API costs: black-box EXECUTIVEs still require API access, which can be expensive at high call volumes.

Why it matters

MEMO reframes memory as an independently trainable, deployable, and swappable module — analogizing the move from coprocessor to standalone compute unit. One MEMORY model can serve many EXECUTIVE models, letting enterprises internalize a knowledge base once and call it from any LLM. And if a 1.5B model can memorize a full corpus, personal knowledge bases could become tiny models attached to Claude or GPT — no vector DB, no retrieval pipeline, no chunk-boundary semantic breaks.

Conclusion

MEMO is not a RAG replacement but a different route: knowledge is invoked at inference time rather than pulled at retrieval time. It has structural advantages for cross-document synthesis, retrieval-noise robustness, and multi-EXECUTIVE support; RAG remains more practical for pure retrieval precision, cost-sensitive, or out-of-distribution query scenarios. Still, the "memory as a model" paradigm is a clear milestone in knowledge-injection architecture.

> References > - arXiv: 2605.15156 — Quek et al., "MEMO: Memory as a Model", May 2026 > - Marktechpost coverage: https://www.marktechpost.com/2026/05/26/memo-a-modular-framework/ > - NeurIPS 2023 LongMem paper (related work on decoupled memory)

Tags

#memo#llm-memory#rag#knowledge-injection#model-merging#multi-hop-reasoning#paper-review#long-context

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208288