MeMo: Memory as a Model — Training a Second Brain for Frozen LLMs

Forum topic · 小凯 · 2026-05-17

Summary

MeMo (arXiv: 2605.15156), from researchers at NUS, MIT CSAIL, A*STAR and partners, introduces a plug-and-play approach to knowledge integration that keeps the base LLM completely frozen. Instead of stuffing documents into context (RAG/ICL) or fine-tuning parameters (risking catastrophic forgetting), MeMo trains a smaller Memory Model on synthetic Reflection QA data distilled from the corpus via a five-step pipeline: fact extraction, consolidation, verification/rewriting, entity surfacing (countering the Reversal Curse), and cross-document synthesis. At inference, the Executive Model interacts with the Memory Model through a three-phase protocol (grounding, entity identification, answer synthesis) via a black-box I/O interface, making MeMo compatible with any open or proprietary LLM. On BrowseComp-Plus, NarrativeQA and MuSiQue, MeMo outperforms BM25, NV-Embed-V2, HippoRAG2 and Cartridges in most settings, and is remarkably robust to retrieval noise. Model merging (e.g., TIES) enables cheap incremental knowledge updates. The method offers a decoupled, transferable, constant-cost memory layer for LLMs.

MeMo: Memory as a Model — A Community Discussion Digest

This post is a detailed Chinese-language walkthrough of the paper MeMo: Memory as a Model (arXiv: 2605.15156) by Ryan Wei Heng Quek, Sanghyuk Lee, et al., from NUS, MIT CSAIL, A*STAR, University of Tokyo, AI Singapore, and SMART (cs.CL, cs.AI, cs.LG). Below is a structured English summary of the argument and findings.

Key points

The problem: After pre-training, LLM knowledge is frozen. Existing fixes each have hard limits:
*In-context learning* — quadratic cost growth with corpus size, degraded performance on large document sets.
*RAG* — sensitive to retrieval noise; struggles to synthesize clues spread across many documents.
*Fine-tuning* — expensive, risks catastrophic forgetting and safety-alignment erosion, and is impossible for closed-source models.
*Latent-memory methods* (AutoCompressor, Gist Tokens, ICAE) — compressed representations are tightly bound to the encoder model, so they cannot transfer across LLMs.
The core idea: Train an independent, smaller Memory Model (14B vs. a 32B Executive Model in the paper) that internalizes corpus knowledge as parameters, and pair it with the Executive Model through a black-box input-output interface. No retrieval index, no base-LLM modification, no catastrophic forgetting, constant-size memory, and cross-LLM transferability.

The five-step data synthesis pipeline

A Generator LLM converts the corpus into Reflection QA data:

1. Fact extraction — per-chunk direct facts and inferred (indirect) facts. 2. Consolidation — merge related QA pairs into composite questions, weaving facts into coherent structures. 3. Verification & rewriting — drop or rewrite QA pairs that are not self-contained (unresolved pronouns, implicit references). 4. Entity surfacing — questions describe an entity's attributes; the answer reveals its identity, countering the Reversal Curse. 5. Cross-document synthesis — exploit converging clues and parallel properties across topic-related documents for comparative and multi-hop questions.

Training is plain SFT on these QA pairs, with the crucial constraint that the Memory Model never sees the original documents — forcing genuine parametric internalization rather than context-copying.

Inference: a three-act protocol

1. Grounding — the Executive Model decomposes the query into atomic sub-questions answered independently by the Memory Model. 2. Entity identification — iterative narrowing until a single entity is identified (or budget exhausted). 3. Answer seeking & synthesis — additional supporting facts are collected and synthesized into the final answer.

All returned snippets are compact natural language, so query cost is independent of corpus size, and the Executive Model is treated fully as a black box (no weights, gradients, or logits needed).

Continual integration via model merging

Instead of retraining on the union of corpora (3X cost), MeMo merges task vectors from separately trained Memory Models using methods like TIES (sparsification density ρ=0.3), which performed best on NarrativeQA at near-zero extra compute.

Benchmark results

Evaluated on BrowseComp-Plus (300 questions / 3,541 docs), NarrativeQA (293 questions / 103 docs), and MuSiQue (1,000 questions / 5,296 docs), with Qwen2.5-32B and Gemini-3-Flash as Executive Models:

| Method | BrowseComp-Plus (Qwen / Gemini) | NarrativeQA | MuSiQue | |---|---|---|---| | Perfect Retrieval (upper bound) | 79.67 / 88.33 | 51.42 / 60.41 | 62.83 / 73.00 | | BM25 | 1.11 / 27.00 | 10.24 / 14.33 | 20.00 / 23.20 | | NV-Embed-V2 | 50.67 / 57.00 | 20.59 / 26.62 | 37.47 / 46.60 | | HippoRAG2 | 56.11 / 66.33 | 21.39 / 23.21 | 42.17 / 57.00 | | Cartridges | 0.00 / – | 3.75 / – | 8.57 / – | | MeMo | 54.22 / 66.67 | 26.85 / 53.58 | 48.30 / 60.20 |

MeMo leads decisively on NarrativeQA and MuSiQue (multi-hop), and beats HippoRAG2 with Gemini-3-Flash on BrowseComp-Plus.
Noise robustness: adding distractor documents equal to the evidence set, NV-Embed-V2 and HippoRAG2 drop ~5–6 points, while MeMo *gains* 0.55 points on BrowseComp-Plus and drops only 1.77 points on MuSiQue.
Ablations: larger Memory Models help (1.5B vs 14B), but performance is insensitive to architecture family (Qwen2.5, Gemma3, LFM2.5 at ~1–2B scale).

Why it matters

The post closes with three takeaways: decoupling (memory storage separated from reasoning, both evolving independently), distillation (the synthesis pipeline converts raw information into a higher-order knowledge representation), and resilience (robustness to noise and architecture, plus cheap continual updates via merging). MeMo points toward composable AI systems — a reasoning engine plus swappable memory modules — rather than a single monolithic model.