MeMo: Memory as a Model — External 'Second Brain' for LLM Knowledge Integration

Forum topic · 小凯 · 2026-05-17

Summary

This post analyzes the paper 'MeMo: Memory as a Model' (arXiv: 2605.15156) by researchers from the National University of Singapore, MIT CSAIL, A*STAR, and collaborators. MeMo introduces a plug-and-play Memory Model trained to internalize a document corpus as 'Reflection QA' data, paired with a frozen black-box Executive Model (LLM). Unlike in-context learning, RAG, or fine-tuning, MeMo requires no access to LLM weights, avoids catastrophic forgetting, resists retrieval noise, and keeps query cost independent of corpus size. The post explains the five-step data synthesis pipeline (fact extraction, consolidation, verification/rewriting, entity surfacing, cross-document synthesis), continual knowledge integration via model merging (TIES with density 0.3 performing best), a three-stage multi-turn inference protocol (Grounding, Entity Identification, Answer Seeking & Synthesis), and benchmark results: MeMo outperforms BM25, NV-Embed-V2, HippoRAG2, and Cartridges on NarrativeQA (26.85%/53.58%) and MuSiQue, and is nearly immune to distractor documents, unlike retrieval baselines that drop several points. The author frames MeMo as a modular 'model ecosystem' approach built on decoupling, distillation, and resilience.

MeMo: Memory as a Model — A 'Second Brain' for LLMs

This forum post presents an in-depth walkthrough of the paper MeMo: Memory as a Model (arXiv: 2605.15156), authored by Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, and Armando Solar-Lezama (NUS, MIT CSAIL, A*STAR, University of Tokyo, AI Singapore, SMART).

The Problem with Existing Knowledge-Integration Methods

LLMs are frozen at their pre-training knowledge snapshot. Current workarounds each have hard limits:

In-Context Learning / RAG (non-parametric): limited context windows, quadratic cost growth, high sensitivity to retrieval noise, and inability to synthesize information scattered across many documents.
Fine-tuning (parametric): expensive, risks catastrophic forgetting and erosion of safety alignment, and is impossible for closed-source models.
Latent memory methods (AutoCompressor, Gist Tokens, ICAE, Memorizing Transformers, kNN-LM): compressed representations are tightly bound to the encoder model, so memories cannot transfer across LLMs.

MeMo's design principle: instead of modifying the LLM, train an independent Memory Model that can restate the corpus in clean, self-contained natural language. It is the only approach satisfying all six properties: frozen base LLM, no retrieval index needed, black-box LLM compatible, no catastrophic forgetting, constant-size memory, and cross-LLM transferability.

The Five-Step Data Synthesis Pipeline

A Generator LLM converts the corpus into Reflection QA pairs:

1. Fact Extraction — direct facts plus inferred/indirect facts per chunk. 2. Consolidation — merge related QA pairs into composite ones, teaching coherent knowledge structures. 3. Verification & Rewriting — keep only self-contained QA pairs; rewrite or discard ambiguous ones (unresolved pronouns, implicit references). 4. Entity Surfacing — questions describe entity attributes; answers reveal identity, directly countering the Reversal Curse. 5. Cross-Document Synthesis — build QA from converging clues and parallel properties across related documents.

The resulting datasets are combined as Q_final = Q_ver ∪ Q_ent ∪ Q_cross.

Training and Continual Integration

The Memory Model (14B, smaller than the 32B Executive Model) is SFT-trained to answer questions without ever seeing the source documents, forcing true parametric knowledge internalization rather than context copying. For new corpora, instead of retraining, MeMo merges independently trained Memory Models via task vectors (φ_merged = Merge(φ_0, {τ_i})), with TIES merging (sparsity density ρ = 0.3) performing best on NarrativeQA — at near-zero cost versus full retraining.

Inference: Three-Act Protocol

The Executive Model interacts with the Memory Model through a structured multi-turn protocol: Grounding (decompose query into atomic sub-questions), Entity Identification (iteratively narrow to a single entity), and Answer Seeking & Synthesis. All exchanged messages are compact natural-language snippets whose length is independent of corpus size, and everything works through a standard input-output interface — no weights, gradients, or logits needed.

Benchmark Results

Evaluated on BrowseComp-Plus, NarrativeQA, and MuSiQue with Qwen2.5-32B and Gemini-3-Flash:

| Method | NarrativeQA (Qwen / Gemini) | MuSiQue (Qwen / Gemini) | BrowseComp-Plus (Gemini) | |---|---|---|---| | BM25 | 10.24 / 14.33 | 20.00 / 23.20 | 27.00 | | NV-Embed-V2 | 20.59 / 26.62 | 37.47 / 46.60 | 57.00 | | HippoRAG2 | 21.39 / 23.21 | 42.17 / 57.00 | 66.33 | | MeMo | 26.85 / 53.58 | 48.30 / 60.20 | 66.67 |

Key findings:

MeMo leads on NarrativeQA and MuSiQue, confirming strong long-document comprehension and multi-hop reasoning.
On BrowseComp-Plus it matches or beats HippoRAG2, trailing only when answers live in raw documents rather than internalizable knowledge.
Noise robustness: adding distractor documents equal to the evidence set barely affects MeMo (−1.77 points on MuSiQue, +0.55 on BrowseComp-Plus), while NV-Embed-V2 and HippoRAG2 drop 4.8–6.2 points.
Ablations: larger Memory Models help (1.5B vs 14B), but performance is insensitive to architecture family (Qwen2.5, Gemma3, LFM2.5), showing the parametric knowledge compression generalizes across models.

Why It Matters

The author highlights three themes: decoupling (knowledge storage vs. reasoning, independently upgradable), distillation (corpus refined into structured Reflection QA), and resilience (robust to noise, architecture-agnostic, incrementally updatable). MeMo points toward a future of composable AI systems — an ecosystem of specialized models rather than a single monolithic LLM.