MeMo: Memory as a Model — A Frozen LLM Plus a Trained 'Second Brain' for Knowledge Integration

Forum topic · 小凯 · 2026-05-17

Summary

MeMo (Memory as a Model, arXiv: 2605.15156) is a framework from NUS, MIT CSAIL, A*STAR, and collaborators that integrates new knowledge into large language models without touching their parameters. Instead of in-context learning, RAG, or fine-tuning, MeMo trains a smaller Memory Model (14B, paired with a 32B Executive Model) on synthesized Reflection QA data generated through a five-step pipeline: fact extraction, consolidation, verification/rewriting, entity surfacing (countering the Reversal Curse), and cross-document synthesis. At inference, a structured three-phase protocol (Grounding, Entity Identification, Answer Seeking & Synthesis) lets the Executive Model query the Memory Model via black-box I/O, keeping retrieval cost constant regardless of corpus size. On NarrativeQA, MuSiQue, and BrowseComp-Plus, MeMo outperforms BM25, NV-Embed-V2, HippoRAG2, and Cartridges with Gemini-3-Flash, and shows near-total robustness to retrieval noise (performance actually improved by 0.55 points on BrowseComp-Plus when distractors were added). Model merging (e.g., TIES, density 0.3) enables cheap continual knowledge updates without full retraining. MeMo works with open and proprietary LLMs alike, offering a modular, transferable alternative to RAG and fine-tuning.

MeMo: Memory as a Model — Forging a Second Brain Instead of Stuffing Context

Paper: MeMo: Memory as a Model (arXiv: 2605.15156) Authors: Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, Armando Solar-Lezama Institutions: National University of Singapore, MIT CSAIL, A*STAR, University of Tokyo, AI Singapore, SMART Categories: cs.CL, cs.AI, cs.LG

Key points

Core idea: Rather than modifying a frozen LLM, MeMo attaches a separately trained Memory Model that has internalized a corpus as parameterized knowledge, paired with the main LLM (the Executive Model) via a black-box, natural-language interface.

Limitations of existing approaches

Non-parametric (ICL, RAG): context windows saturate; costs grow with corpus size; RAG is sensitive to retrieval noise and struggles with multi-document synthesis.
Parametric (continual pre-training, SFT): expensive, risks catastrophic forgetting, can erode safety alignment (Qi et al., 2024), and is impossible for closed models.
Latent-memory methods (AutoCompressor, Gist Tokens, ICAE): compressed representations are bound to the encoder model and not transferable across LLMs.
MeMo is claimed to be the only method that simultaneously keeps the base LLM frozen, needs no retrieval index, supports black-box LLMs, avoids catastrophic forgetting, keeps memory size constant, and transfers across LLMs.

Five-step Reflection QA synthesis pipeline

1. Fact extraction: direct and indirect QA pairs per chunk. 2. Consolidation: merge related QA pairs into compositional ones. 3. Verification & rewriting: enforce self-contained questions/answers (resolve pronouns, implicit references); discard failures. 4. Entity surfacing: descriptions point to entities, combating the Reversal Curse. 5. Cross-document synthesis: converging clues and parallel properties across related documents.

Training

The Memory Model (14B, smaller than the 32B Executive Model) is fine-tuned on the QA sets while never seeing the source documents, forcing genuine internalization:

\mathcal{L}(\varphi) = -\sum_{(q_i, a_i) \in \mathcal{Q}_{\text{final}}} \sum_{t=1}^{|a_i|} \log \mathcal{M}_\varphi\left(a_i^{(t)} \mid q_i, a_i^{(1:t-1)}\right)

Continual integration via model merging

Task vectors $\tau_i = \varphi_i - \varphi_0$ from separate corpora are merged (14 configurations tested; TIES with density ρ=0.3 worked best on NarrativeQA) — far cheaper than retraining.

Three-phase inference protocol

1. Grounding: the Executive Model decomposes the query into atomic sub-questions. 2. Entity identification: iterative narrowing to a single entity e*. 3. Answer seeking & synthesis: gather supporting facts, then synthesize the final answer. All exchanges are compact natural-language snippets, so query cost is constant regardless of corpus size; no access to weights, gradients, or logits is required.

Results

| Method | BrowseComp-Plus (Gemini-3-Flash) | NarrativeQA (Gemini-3-Flash) | MuSiQue (Gemini-3-Flash) | |---|---|---|---| | Perfect Retrieval (upper bound) | 88.33 | 60.41 | 73.00 | | BM25 | 27.00 | 14.33 | 23.20 | | NV-Embed-V2 | 57.00 | 26.62 | 46.60 | | HippoRAG2 | 66.33 | 23.21 | 57.00 | | MeMo | 66.67 | 53.58 | 60.20 |

MeMo leads on NarrativeQA and MuSiQue; matches or beats HippoRAG2 on BrowseComp-Plus with Gemini-3-Flash.
Noise robustness: with distractor documents added, NV-Embed-V2 and HippoRAG2 dropped ~6 points on BrowseComp-Plus while MeMo improved by 0.55 points; on MuSiQue MeMo dropped only 1.77 points.
Ablations: larger Memory Models help (task-dependent); performance is insensitive to Memory Model architecture (Qwen2.5 vs Gemma3 vs LFM2.5 at ~1–2B).

Why it matters

Decoupling: knowledge storage and reasoning evolve independently; domain-specific Memory Models can be swapped in at inference.
Distillation: the pipeline converts raw text into higher-order, structured Reflection QA knowledge.
Resilience: robustness to noise, architecture-agnostic memory, and cheap continual updates suit messy real-world corpora.

MeMo points toward modular AI ecosystems — a reasoning engine plus a dedicated, always-updatable memory module — as a pragmatic third path between RAG and fine-tuning.