MeMo: Memory as a Model - Training a Dedicated 'Second Brain' Instead of Stuffing Contexts

Forum topic · 小凯 · 2026-05-17

Summary

MeMo (Memory as a Model, arXiv 2605.15156) is a knowledge-integration framework from NUS, MIT CSAIL, A*STAR, and partners that pairs any frozen LLM with a separately trained Memory Model. A five-step data synthesis pipeline (fact extraction, consolidation, verification/rewriting, entity surfacing, cross-document synthesis) converts a corpus into self-contained Reflection QA pairs, which are distilled into the Memory Model's parameters via supervised fine-tuning. At inference, the LLM (Executive Model) queries the Memory Model through a structured three-stage protocol: grounding sub-questions, iterative entity identification, and answer synthesis. Because the memory model works through a black-box text interface, it is compatible with open and proprietary LLMs, immune to catastrophic forgetting, and its retrieval cost is independent of corpus size. On BrowseComp-Plus, NarrativeQA, and MuSiQue, MeMo outperforms BM25, NV-Embed-V2, HippoRAG2, and Cartridges baselines with Qwen2.5-32B and Gemini-3-Flash, and shows remarkable robustness to retrieval noise. Model merging (TIES) enables cheap incremental knowledge updates.

MeMo: Memory as a Model - Not Stuffing Contexts, but Distilling a Second Brain

Paper: MeMo: Memory as a Model (arXiv: 2605.15156) Authors: Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, Armando Solar-Lezama Institutions: National University of Singapore, MIT CSAIL, A*STAR, University of Tokyo, AI Singapore, SMART Categories: cs.CL, cs.AI, cs.LG

The Core Problem

After pre-training, an LLM's knowledge is frozen at a snapshot in time. Existing workarounds all have serious flaws:

In-Context Learning / RAG: context windows are finite, costs grow with corpus size, and retrieved noise can mislead the model. RAG also struggles when answers require synthesizing information spread across many documents.
Fine-tuning / continual pre-training: expensive, risks catastrophic forgetting and erosion of safety alignment, and is impossible for closed-source models like GPT-4 or Gemini.
Latent-memory methods (AutoCompressor, Gist Tokens, ICAE): compressed representations are tightly coupled to the encoder model that produced them and cannot transfer across LLMs.

MeMo's answer: train an independent Memory Model that internalizes the corpus and answers queries in clean natural language, so the main LLM (Executive Model) never changes its parameters.

| Property | Non-parametric | Parametric | Latent memory | MeMo | |---|---|---|---|---| | Frozen base LLM | ✅ | ❌ | ✅ | ✅ | | No retrieval index | ❌ | ✅ | ✅ | ✅ | | Black-box LLM compatible | ✅ | ❌ | ❌ | ✅ | | No catastrophic forgetting | ✅ | ❌ | ✅ | ✅ | | Constant-size memory | ❌ | ✅ | ✅ | ✅ | | Transferable across LLMs | ✅ | ❌ | ❌ | ✅ |

MeMo is the only approach that satisfies all six.

The Five-Step Synthesis Pipeline

A Generator LLM transforms the corpus into Reflection QA pairs:

1. Fact Extraction - direct facts plus indirect inferences per chunk. 2. Consolidation - merging related QA pairs into composite questions, forcing the model to weave facts into coherent structures. 3. Verification & Rewriting - each QA pair must be self-contained (no unresolved pronouns or implicit references); failures are rewritten or discarded. 4. Entity Surfacing - questions describe an entity's attributes, answers reveal its identity, combating the "Reversal Curse" with bidirectional reasoning. 5. Cross-Document Synthesis - generating questions from converging clues and parallel properties across related documents, enabling multi-document reasoning.

Training the Memory Model

The Memory Model (14B parameters, vs. a 32B Executive Model) is fine-tuned on the QA set while never seeing the original documents:

\mathcal{L}(\varphi) = -\sum_{(q_i, a_i) \in \mathcal{Q}_{\text{final}}} \sum_{t=1}^{|a_i|} \log \mathcal{M}_\varphi\left(a_i^{(t)} \mid q_i, a_i^{(1:t-1)}\right)

This forces genuine parameterized knowledge internalization rather than a copy-from-context shortcut.

Continual updates via model merging

Instead of retraining on $\mathcal{D}_1 \cup \mathcal{D}_2$ , MeMo merges task vectors $\tau_i = \varphi_i - \varphi_0$ back onto the base model. TIES merging (density ρ=0.3) performed best on NarrativeQA, at near-zero extra cost versus full retraining.

Inference: A Three-Act Protocol

The Executive Model interacts with the Memory Model through:

1. Grounding - decompose the query into atomic sub-questions answered independently. 2. Entity Identification - iteratively narrow candidate entities to a single target e*. 3. Answer Seeking & Synthesis - collect supporting facts and synthesize the final answer:

\hat{a} = \mathcal{M}_\theta\left(q, \{m_k\}_{k=1}^K, e^*, m_{\text{seek}}\right)

All responses are compact natural-language snippets of size independent of the corpus, and the Executive Model is treated as a black box (no weights, gradients, or logits needed).

Experimental Results

Evaluated on BrowseComp-Plus (300 questions, 3,541 documents), NarrativeQA (293 questions, 103 documents), and MuSiQue (1,000 questions, 5,296 documents):

| Method | BC-P (Qwen) | BC-P (Gemini) | NQA (Qwen) | NQA (Gemini) | MuSiQue (Qwen) | MuSiQue (Gemini) | |---|---|---|---|---|---|---| | BM25 | 1.11 | 27.00 | 10.24 | 14.33 | 20.00 | 23.20 | | NV-Embed-V2 | 50.67 | 57.00 | 20.59 | 26.62 | 37.47 | 46.60 | | HippoRAG2 | 56.11 | 66.33 | 21.39 | 23.21 | 42.17 | 57.00 | | Cartridges | 0.00 | - | 3.75 | - | 8.57 | - | | MeMo | 54.22 | 66.67 | 26.85 | 53.58 | 48.30 | 60.20 |

Key findings:

NarrativeQA: MeMo dramatically outperforms retrieval baselines, having internalized long-range character relationships during training.
MuSiQue: clear lead on multi-hop reasoning.
Noise robustness: when distractor documents equal in number to the evidence are added, NV-Embed-V2 drops 6.22 points on BrowseComp-Plus and HippoRAG2 drops 6.22 points, while MeMo *gains* 0.55 points and drops only 1.77 on MuSiQue - distilled Reflection QA filters out noise that plagues retrieval.
Ablations: larger Memory Models help (task-dependent), and performance is insensitive to memory-model architecture (Qwen2.5 vs. Gemma3 vs. LFM2.5 at 1-2B scale).

Why It Matters

1. Decoupling - knowledge storage and knowledge use are separated; either component can be upgraded independently, with per-domain memory models callable on demand. 2. Distillation - the pipeline converts raw information into a higher-order, denoised knowledge representation. 3. Resilience - constant-cost retrieval, black-box compatibility, and merge-based updates work in a messy real-world corpus environment.

MeMo points toward future AI systems as a model ecosystem - a reasoning engine, a memory module, and interfaces collaborating over standard protocols - rather than one monolithic model.

> "We treat the Executive Model as a black box without accessing its weights, gradients, or output logits, which allows MeMo to be plug-and-play with any LLM, including open-source and proprietary closed-source models."

*Deep-dive based on arXiv: 2605.15156, originally published on zhichai.net.*