MEMO: Unfreezing LLM Knowledge with a Small Companion Model, Not Fine-tuning

Forum topic · 小凯 · 2026-05-26

Summary

MEMO (Memory as a Model) is a new architecture that updates a frozen large language model without fine-tuning or retrieval. It pairs a frozen EXECUTIVE model (e.g., Qwen2.5-32B) with a small, separately trained MEMORY model (1.5B–14B) that stores knowledge as parameterized QA pairs synthesized from documents via a five-step pipeline, including critical cross-document synthesis. At inference, a three-stage multi-turn protocol lets the EXECUTIVE query the MEMORY model as a black box, keeping it compatible with proprietary APIs. On NarrativeQA and MuSiQue, MEMO-14B outperforms strong RAG baselines like HippoRAG2, and it is nearly immune to retrieval noise since it relies on parametric memory rather than retrieved documents. Model merging (TIES) enables incremental updates, cutting compute from Θ(K²) to Θ(K) with some accuracy loss. The post reviews the architecture, training objective, ablations, limitations, and deployment implications.

After pre-training, a large language model's knowledge is effectively "frozen." The industry's two mainstream fixes each have drawbacks:

RAG — stuff documents into context. As documents pile up, noise explodes, and cross-document relationships are hard to capture.
Fine-tuning — modify model weights directly. It's compute-hungry and triggers "catastrophic forgetting," scrambling old knowledge along with new.

MEMO (Memory as a Model) takes a third path: don't touch the big model — give it a dedicated "memory chip."

---

Core Architecture: A Two-Model Division of Labor

| Component | Role | Size | Status | |:---|:---|:---|:---| | EXECUTIVE | Reasoning and answering | 32B (e.g., Qwen2.5-32B) | Frozen | | MEMORY | Knowledge storage and retrieval | 1.5B–14B (e.g., Qwen2.5-14B) | Trained |

Flow: user query → EXECUTIVE decides what knowledge is needed → queries MEMORY → MEMORY returns parameterized knowledge → EXECUTIVE synthesizes the answer.

Key design: MEMORY's response length is independent of corpus size. Whether the backing store is a thousand or ten thousand documents, MEMORY outputs refined knowledge fragments — fundamentally different from RAG, which stuffs raw documents into context.

---

Data Alchemy: Five Steps from Documents to "Reflections"

MEMORY doesn't read raw documents; it trains on distilled "reflective QA pairs." The synthesis pipeline has five steps:

| Step | Name | Core operation | |:---|:---|:---| | 1 | Fact extraction | Direct extraction of explicit facts + indirect extraction of inferred/synthesized information | | 2 | Integration | Identify QA pairs sharing context; merge into multi-fact questions | | 3 | Validation & rewrite | Check self-containment — no coreference resolution, no implicit references | | 4 | Entity emergence | Generate "entity–attribute–relation" QA pairs where answers reveal entity identity | | 5 | Cross-document synthesis | Identify converging clues and parallel attributes across topically related document groups |

Step 5 is the most critical. Ablations show removing it crashes NarrativeQA accuracy from 24.0% to 6.37%. Step 4 specifically mitigates the reversal curse — a model that knows "Obama is the US president" failing to answer "who is the US president?"

One iron rule: generated QA pairs never embed document identifiers or watermarks, preventing MEMORY from taking shortcuts like guessing answers from document IDs.

---

Training Objective: Memorize Answers, Not Source Text

MEMORY's loss counts only answer tokens:

\mathcal{L}(\phi) = -\sum_{(q_i, a_i)} \sum_{t=1}^{|a_i|} \log M_\phi\left(a_i^{(t)} \mid q_i, a_i^{(<t)}\right)

Key constraint: conditioning depends only on the question and prior answer tokens, never on the source document. This forces MEMORY to parameterically internalize knowledge rather than learn a "copy from context" pattern.

Training is standard: AdamW, lr=2e-5, 3 epochs, BF16, Flash Attention 2. The value lies in architecture, not training tricks.

---

Inference: A Three-Stage Multi-Turn Protocol

Stage 1: Grounding — decompose the query into atomic probing sub-questions; MEMORY answers each independently.

Stage 2: Entity identification — iteratively narrow candidate entities using Stage 1 responses until convergence or budget exhaustion.

Stage 3: Answer synthesis — query additional supporting facts for confirmed entities; synthesize all responses into a final answer.

Different prompts, temperatures, and budgets per stage. All interaction goes through input–output interfaces, so EXECUTIVE is a black box — MEMO works with proprietary APIs.

---

Results: Immune to Retrieval Noise

| Method | BrowseComp-Plus | NarrativeQA | MuSiQue | |:---|:---|:---|:---| | Perfect Retrieval (ceiling) | 79.67% | 51.42% | 62.83% | | BM25 | 1.11% | 10.24% | 20.00% | | NV-Embed-V2 | 50.67% | 20.59% | 37.47% | | HippoRAG2 | 56.11% | 21.39% | 42.17% | | MEMO (14B) | 54.22% | 26.85% | 48.30% |

NarrativeQA is the most telling result: long-document QA over whole books. MEMO beats all RAG baselines by +5.46% over HippoRAG2.

On BrowseComp-Plus, MEMO slightly trails HippoRAG2 (54.22% vs 56.11%) — answers live entirely in the raw documents, which favors direct retrieval. MEMO wins when answers require cross-document synthesis.

The hardest result: noise robustness. With retrieval noise, NV-Embed-V2 and HippoRAG2 each drop ↓6.22, while MEMO actually rises ↑0.55 (53.67% → 54.22%). MEMO doesn't retrieve — its knowledge is parametric.

---

How Big Should MEMORY Be?

| MEMORY size | BrowseComp-Plus | NarrativeQA | MuSiQue | |:---|:---|:---|:---| | 1.5B | 44.11% | 24.00% | 42.90% | | 14B | 54.22% | 26.85% | 48.30% |

1.5B already works well; 14B is better. Experiments with other architectures (Gemma3-1B, LFM2.5-1.2B) suggest scale matters more than pretraining lineage.

---

Incremental Updates: Model Merging Saves 33% Compute

MEMO supports model merging: train per-corpus task vectors τᵢ = φᵢ − φ₀ and merge with TIES, reducing compute from Θ(K²) to Θ(K) — 33% savings with two corpora (48h vs 72h), 5.5× with ten (1320h vs 240h).

The cost is accuracy: merged NarrativeQA drops from 26.85% to 15.81% (↓11.04%). Use merging for frequent updates; full retraining for maximum precision.

---

MEMO Is Not a Silver Bullet

1. BrowseComp-Plus weakness — when answers depend on verbatim document content, RAG's direct access wins. 2. Merging loses accuracy — better merging algorithms are needed. 3. Data synthesis cost — ~1.6M QA pairs for NarrativeQA; the upfront compute depends on update frequency. 4. EXECUTIVE dependence — a stronger EXECUTIVE (Gemini-3-Flash vs Qwen2.5-32B) extracts up to +26.73% more on NarrativeQA.

---

Why MEMO Matters

| Property | Non-parametric (RAG) | Parametric (fine-tuning) | Latent memory | MEMO | |:---|:---|:---|:---|:---| | Frozen base LLM | ✓ | ✗ | ✓ | ✓ | | No retrieval index | ✗ | ✓ | ✓ | ✓ | | Black-box compatible | ✓ | ✗ | ✗ | ✓ | | No catastrophic forgetting | ✓ | ✗ | ✓ | ✓ | | Constant-size memory | ✗ | ✓ | ✗ | ✓ | | Transferable across LLMs | ✓ | ✗ | ✗ | ✓ |

For deployment this means: no retraining the big model, no retrieval index to maintain, no context-length limits, no forgetting, and compatibility with proprietary APIs like Claude/GPT-4.

One-line summary: MEMO's idea isn't "make the big model learn more" but "give the big model a small companion that learns." Knowledge updating becomes an incrementally maintainable engineering problem instead of a destructive one.

---

References

Chang et al. (2025). MEMO: Memory as a Model. arXiv:2605.15156
GitHub: https://github.com/microsoft/MEMO