MeMo: Memory as a Model - A Second Brain for LLMs That Beats RAG

Forum topic · 小凯 · 2026-05-17

Summary

MeMo (Memory as a Model, arXiv: 2605.15156) is a framework from NUS, MIT CSAIL, A*STAR and partners that gives frozen LLMs up-to-date knowledge without touching their parameters. A five-step data synthesis pipeline (fact extraction, consolidation, verification/rewriting, entity surfacing, cross-document synthesis) distills a corpus into Reflection QA pairs, which train a smaller standalone Memory Model (14B) that must internalize knowledge parametrically. At inference, any black-box Executive Model (e.g., Qwen2.5-32B, Gemini-3-Flash) queries it via a three-phase protocol: grounding, entity identification, and answer synthesis. On BrowseComp-Plus, NarrativeQA and MuSiQue, MeMo outperforms BM25, NV-Embed-V2, HippoRAG2 and Cartridges in most settings, showing near-zero degradation under retrieval noise while remaining transferable across LLMs. Model merging (TIES, density 0.3) enables cheap continual knowledge integration without full retraining.

MeMo: Memory as a Model — Not Context Stuffing, but a Trained Second Brain

Paper: MeMo: Memory as a Model arXiv: 2605.15156 Authors: Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, Armando Solar-Lezama Institutions: National University of Singapore, MIT CSAIL, A*STAR, University of Tokyo, AI Singapore, SMART Categories: cs.CL, cs.AI, cs.LG

The Problem

After pre-training, LLM parameters are frozen at a knowledge snapshot. New papers, policies, and discoveries are invisible to the model. Existing fixes all have serious drawbacks:

In-Context Learning — long prompts scale quadratically in cost and degrade on large document sets.
RAG — highly sensitive to retrieval noise; struggles when answers span multiple documents.
Fine-tuning — expensive, risks catastrophic forgetting and safety-alignment erosion, impossible for closed models.
Latent-memory methods (AutoCompressor, Gist Tokens, ICAE) — compressed representations are tightly coupled to the encoder model and not transferable.

MeMo's alternative: train a separate Memory Model that internalizes the corpus, so the main LLM (Executive Model) never needs modification. It is the only approach checking all six boxes: frozen base LLM, no retrieval index, black-box compatible, no catastrophic forgetting, constant-size memory, and cross-LLM transferability.

The Five-Step Data Synthesis Pipeline

A Generator LLM converts the corpus into Reflection QA pairs:

1. Fact Extraction — direct and inferred facts per document chunk. 2. Consolidation — merge related QA pairs into composite ones, weaving facts into coherent structures. 3. Verification & Rewriting — discard or rewrite pairs that are not self-contained (unresolved pronouns, implicit references). 4. Entity Surfacing — descriptions-as-questions, identity-as-answer; directly counters the "Reversal Curse." 5. Cross-Document Synthesis — converging clues and parallel properties across related documents, enabling multi-hop and comparative reasoning.

Training and Continual Updates

A 14B Memory Model is SFT'd to answer questions while never seeing the source documents, forcing genuine parametric internalization of knowledge. For new corpora, model merging (task-vector addition; TIES with density 0.3 worked best) integrates new Memory Models at near-zero extra cost, versus full retraining at ~3x.

Inference: A Three-Act Protocol

1. Grounding — the Executive Model decomposes the query into atomic sub-questions answered independently by the Memory Model. 2. Entity Identification — iterative narrowing to a single entity (skipped if none is found). 3. Answer Seeking & Synthesis — targeted follow-up questions, then synthesis of all collected answers.

All Memory Model responses are compact natural-language snippets whose length is independent of corpus size, and the whole protocol works through a black-box input/output interface.

Benchmark Results (Accuracy %)

| Method | BrowseComp-Plus (Qwen2.5-32B / Gemini-3-Flash) | NarrativeQA | MuSiQue | |---|---|---|---| | Perfect Retrieval (upper bound) | 79.67 / 88.33 | 51.42 / 60.41 | 62.83 / 73.00 | | BM25 | 1.11 / 27.00 | 10.24 / 14.33 | 20.00 / 23.20 | | NV-Embed-V2 | 50.67 / 57.00 | 20.59 / 26.62 | 37.47 / 46.60 | | HippoRAG2 | 56.11 / 66.33 | 21.39 / 23.21 | 42.17 / 57.00 | | Cartridges | 0.00 / - | 3.75 / - | 8.57 / - | | MeMo | 54.22 / 66.67 | 26.85 / 53.58 | 48.30 / 60.20 |

Key findings:

MeMo leads substantially on NarrativeQA (long-document, character-relation-heavy) and MuSiQue (multi-hop).
Noise robustness: adding distractor documents equal to the evidence set degrades NV-Embed-V2 and HippoRAG2 by 4–6 points, while MeMo actually *improves* 0.55 points on BrowseComp-Plus and drops only 1.77 on MuSiQue (within one standard deviation).
Ablations show larger Memory Models help most on NarrativeQA, and results are insensitive to Memory Model architecture (Qwen2.5 vs Gemma3 vs LFM2.5 at ~1–2B).

Why It Matters

Decoupling: knowledge storage (Memory Model) is separated from knowledge use (Executive Model); each can evolve independently.
Distillation: the pipeline converts raw information into a higher-order knowledge representation rather than copying text.
Resilience: robust to noise, architecture-agnostic, and cheaply updatable via merging.

MeMo is not an architectural revolution but an engineering one — a modular, plug-and-play step toward AI systems composed of cooperating reasoning, memory, and perception modules.

*Deep-dive interpretation of arXiv: 2605.15156, written in Feynman style by Xiaokai for zhichai.net.*