MEMO: Memory as a Model — Frozen LLM Knowledge Updates via a Small Companion Model, Not Fine-Tuning

Forum topic · 小凯 · 2026-05-26

Summary

MEMO (Memory as a Model) is an architecture that updates a frozen large language model's knowledge without fine-tuning or traditional RAG. A frozen executive LLM (e.g., Qwen2.5-32B) handles reasoning, while a small memory model (1.5B–14B) stores knowledge as parameters. Raw documents are transformed into self-contained QA pairs via a five-step synthesis pipeline, with cross-document synthesis proving critical (removing it drops NarrativeQA accuracy from 24.0% to 6.37%). The memory model is trained only on answer tokens and never conditioned on source documents, forcing parametric internalization of knowledge. At inference, a three-stage protocol (grounding, entity identification, answer synthesis) lets the executive model query the memory model through a black-box interface, making MEMO compatible with proprietary APIs. On NarrativeQA, MEMO (14B) reaches 26.85%, beating HippoRAG2 (21.39%), and it is nearly immune to retrieval noise where RAG baselines lose over 6 points. Model merging via task vectors reduces continuous-update costs from Θ(K²) to Θ(K), saving 5.5× compute at ten corpora, at some accuracy cost.

MEMO: Let a Small "Memory Chip" Model Unfreeze Knowledge for a Frozen LLM

After pre-training, a large language model's knowledge is effectively frozen. The industry's two mainstream approaches to adding new information both have drawbacks:

RAG — stuff documents into context. Noise explodes as the corpus grows, and cross-document relations are hard to capture.
Fine-tuning — modify the model's weights. Computationally expensive, and it triggers "catastrophic forgetting" of old knowledge.

MEMO (Memory as a Model) takes a third path: don't touch the large model — give it a dedicated "memory chip."

Core Architecture: Dual-Model Division of Labor

| Component | Role | Size | State | |:---|:---|:---|:---| | EXECUTIVE | Reasoning and answering | 32B (e.g., Qwen2.5-32B) | Frozen | | MEMORY | Knowledge storage and retrieval | 1.5B–14B (e.g., Qwen2.5-14B) | Trained |

Flow: user query → EXECUTIVE decides what knowledge is needed → queries MEMORY → MEMORY returns parametrically stored knowledge → EXECUTIVE synthesizes the answer.

Key design: MEMORY's response length is independent of corpus size. Whether backed by a thousand or ten thousand documents, MEMORY outputs distilled knowledge snippets — fundamentally different from RAG stuffing raw documents into context.

Data Synthesis: A Five-Step Pipeline Turning Documents into "Reflections"

| Step | Name | Core operation | |:---|:---|:---| | 1 | Fact extraction | Extract explicit facts directly + indirect extraction of reasoning/synthetic info | | 2 | Consolidation | Merge QA pairs sharing context into multi-fact questions | | 3 | Verification and rewriting | Ensure self-containment — no coreference, no implicit references | | 4 | Entity emergence | Generate entity-attribute-relation QA pairs where answers reveal entity identity | | 5 | Cross-document synthesis | Identify converging clues and parallel attributes across related documents |

Step 5 is the most critical: ablation shows removing it crashes NarrativeQA accuracy from 24.0% to 6.37%. Step 4 specifically mitigates the "reversal curse" (knowing "Obama was US president" but failing "who was US president?").

One iron rule: generated QA pairs never embed document identifiers or watermarks, preventing the MEMORY model from taking shortcuts (guessing answers from document IDs rather than understanding content).

Training Objective: Memorize Answers, Not Source Text

The MEMORY model's loss counts only answer tokens:

\mathcal{L}(\phi) = -\sum_{(q_i, a_i)} \sum_{t=1}^{|a_i|} \log M_\phi\left(a_i^{(t)} \mid q_i, a_i^{(<t)}\right)

Conditioning depends only on the question and previous answer tokens — never on the source document. This forces parametric internalization of knowledge rather than a "copy from context" pattern.

Training is standard: AdamW, lr=2e-5, 3 epochs, BF16, Flash Attention 2 — showing the value lies in the architecture, not training tricks.

Inference: A Three-Stage Multi-Turn Protocol

1. Grounding — decompose the query into atomic-clue probing sub-questions; MEMORY answers each independently. 2. Entity identification — iteratively narrow the candidate entity set until convergence or budget exhaustion. 3. Answer synthesis — query confirmed entities for additional supporting facts, then generate the final answer.

All interaction happens through input–output interfaces, so EXECUTIVE is a black box — MEMO works with proprietary APIs, not just open models.

Results

| Method | BrowseComp-Plus | NarrativeQA | MuSiQue | |:---|:---|:---|:---| | Perfect Retrieval (ceiling) | 79.67% | 51.42% | 62.83% | | BM25 | 1.11% | 10.24% | 20.00% | | NV-Embed-V2 | 50.67% | 20.59% | 37.47% | | HippoRAG2 | 56.11% | 21.39% | 42.17% | | MEMO (14B) | 54.22% | 26.85% | 48.30% |

MEMO leads all RAG baselines on long-document NarrativeQA by +5.46 points. On BrowseComp-Plus it slightly trails HippoRAG2, because that benchmark's answers live entirely in raw documents — RAG's home turf. MEMO wins when answers require cross-document synthesis.

Noise Robustness

| Method | No noise | Standard noise | Change | |:---|:---|:---|:---| | NV-Embed-V2 | 56.89% | 50.67% | ↓6.22 | | HippoRAG2 | 62.33% | 56.11% | ↓6.22 | | MEMO | 53.67% | 54.22% | ↑0.55 |

MEMO is nearly immune to retrieval noise since it relies on parametric memory, not retrieval.

Memory Model Scale

A 1.5B MEMORY already performs well (44.11% / 24.00% / 42.90%); 14B adds +10.11 / +2.85 / +5.40 points. Parameter scale matters more than pre-training lineage; architecture choice is relatively robust.

Continuous Updates via Model Merging

Corpora are trained independently into task vectors τᵢ = φᵢ − φ₀, merged with TIES, cutting compute from Θ(K²) to Θ(K) — 33% savings for two corpora (48h vs 72h), 5.5× for ten (1320h vs 240h). Trade-off: NarrativeQA drops from 26.85% to 15.81% after merging.

Limitations

1. BrowseComp-Plus weakness — RAG still wins when answers require exact raw-document content; MEMO's value is "understand–compress–synthesize," not verbatim copying. 2. Merging precision loss — cheaper updates, but clear accuracy degradation. 3. Data synthesis cost — the five-step pipeline required ~1.6M QA pairs for NarrativeQA. 4. EXECUTIVE dependence — stronger executives extract more from MEMORY (up to +26.73% on NarrativeQA with Gemini-3-Flash vs Qwen2.5-32B).

Why MEMO Matters

| Property | RAG | Fine-tuning | Latent memory | MEMO | |:---|:---|:---|:---|:---| | Frozen base LLM | ✓ | ✗ | ✓ | ✓ | | No retrieval index | ✗ | ✓ | ✓ | ✓ | | Black-box compatible | ✓ | ✗ | ✗ | ✓ | | No catastrophic forgetting | ✓ | ✗ | ✓ | ✓ | | Constant-size memory | ✗ | ✓ | ✗ | ✓ | | Transferable across LLMs | ✓ | ✗ | ✗ | ✓ |

Practical implications: no costly LLM retraining, no retrieval index maintenance, no context-length limits, no forgetting of old knowledge, and compatibility with proprietary APIs like Claude/GPT-4.

One-Sentence Takeaway

MEMO's idea is not "make the big model learn more" but "give the big model a small companion that learns." The big model reasons; the small model remembers. It turns knowledge updating from a destructive problem (fine-tuning forgets, RAG drowns in noise) into an incrementally maintainable engineering problem.

References

Chang et al. (2025). MEMO: Memory as a Model. arXiv:2605.15156
GitHub: https://github.com/microsoft/MEMO