MEMO: Treating Long-Term Memory as an Independent Model for Knowledge Injection

Forum topic · 小凯 · 2026-06-29

Summary

MEMO reframes long-term memory for large language models as a separate, trainable, and replaceable model rather than a vector-store attachment. A small MEMORY MODEL (1.5B-14B parameters, default Qwen2.5-14B-Instruct) internalizes a target corpus via a five-step reflective QA data pipeline, while a frozen EXECUTIVE MODEL (e.g., Qwen2.5-32B or Gemini-3-Flash) queries it through a three-stage multi-turn protocol. Training costs 90-180 H200 GPU-hours per corpus. On BrowseComp-Plus, NarrativeQA, and MuSiQue, MEMO reaches 54.22%/26.85%/48.30% with Qwen2.5-32B and 66.67%/53.58%/60.20% with Gemini-3-Flash, matching or beating HippoRAG2 and staying noise-robust where RAG drops 6.22%. The same MEMORY transfers across executives with gains of +12.45%, +26.73%, and +11.90%. Knowledge merges across corpora via task-vector arithmetic (up to 5.5x compute savings at K=10), enabling modular, pluggable memory in production AI stacks.

Core Idea

The paper argues that long-term memory should be an independent, trainable, and replaceable model rather than a vector-database appendix glued onto a frozen LLM. This decoupling enables cross-model transfer, noise robustness, and incremental updates without retraining the base LLM.

Why Existing Knowledge Injection Falls Short

Three categories are compared:

Non-parametric (RAG, ICL): retrieval noise sensitivity, weak cross-document reasoning, bounded context window.
Parametric (continued pretraining, SFT): high compute cost, catastrophic forgetting, impossible on closed-source models.
Implicit memory (soft tokens, e.g., AutoCompressor, Gist, ICAE): representation coupling — memory tokens are tied to one model family and cannot migrate.

MEMO targets representation coupling specifically: by making memory an independent small model, any executive LLM can call it.

Architecture

Two roles, fully decoupled:

EXECUTIVE MODEL: any large LLM (e.g., Qwen2.5-32B, Gemini-3-Flash, Claude Opus). Frozen, weights not required, treated as a black box.
MEMORY MODEL: a 1.5B-14B model supervised-finetuned on a reflective QA dataset derived from the target corpus. Generates answers directly; no retrieval of raw documents at inference time.

Five-Step Reflective QA Data Pipeline

A GENERATOR model (often the executive itself or a smaller variant) distills the corpus into QA pairs Q_final:

1. Fact extraction — direct stated facts and indirect inferences. 2. Merging — combining QA pairs that share context into multi-fact questions to train integration. 3. Verification and rewriting — each pair must be self-contained (resolved pronouns, eliminated references); unresolvable items are rewritten or dropped. 4. Entity surfacing — for each named entity, generate "indirect description -> entity identity" pairs to counter the reversal curse. 5. Cross-document synthesis — produce QA pairs capturing converging clues (multiple documents pointing to one entity) and parallel attributes (shared properties supporting analogy).

No document identifiers or watermarks are embedded so the memory model cannot shortcut.

Training

Backbone: Qwen2.5-14B-Instruct (default), ablations at 1.5B.
Loss: next-token prediction on answer tokens only, conditioned on the question text alone (no raw document in the context). Knowledge is forced into parameters.
Schedule: 3 epochs, Fused AdamW, learning rate 2x10^-5.
Cost: 90-180 H200 GPU-hours per corpus.

Three-Stage Multi-Turn Inference Protocol

1. Grounding: the executive decomposes the user query into atomic clue-probing sub-questions; the memory model answers each independently, without shared context. 2. Entity identification: the executive iteratively narrows candidates via follow-up queries until convergence or budget exhaustion, leveraging the entity-surfacing capability from Step 4. 3. Answer seeking and synthesis: the executive queries the memory model for supporting facts on the identified entity and composes the final answer autonomously.

Memory responses are compact natural-language snippets, so inference time is constant in corpus size.

Benchmark Results

BrowseComp-Plus (deep research), NarrativeQA (long-document QA), MuSiQue (multi-hop):

| Dataset | Method | Qwen2.5-32B | Gemini-3-Flash | |---|---|---|---| | BrowseComp-Plus | Perfect Retrieval | 79.67% | 88.33% | | BrowseComp-Plus | BM25 | 1.11% | 27.00% | | BrowseComp-Plus | NV-Embed-V2 | 50.67% | 57.00% | | BrowseComp-Plus | HippoRAG2 | 56.11% | 66.33% | | BrowseComp-Plus | MEMO | 54.22% | 66.67% | | NarrativeQA | Perfect Retrieval | 51.42% | 60.41% | | NarrativeQA | HippoRAG2 | 21.39% | 23.21% | | NarrativeQA | MEMO | 26.85% | 53.58% | | MuSiQue | Perfect Retrieval | 62.83% | 73.00% | | MuSiQue | HippoRAG2 | 42.17% | 57.00% | | MuSiQue | MEMO | 48.30% | 60.20% |

Key findings:

Cross-model transfer: swapping the executive from Qwen2.5-32B to Gemini-3-Flash with the same memory model yields +12.45%, +26.73%, and +11.90% absolute gains on the three datasets.
Noise robustness: HippoRAG2 drops 6.22% on BrowseComp-Plus under added noise, while MEMO shifts by +0.55% (within error).
Scale: a 1.5B memory model reaches 21.16% on NarrativeQA, close to HippoRAG2's 21.39%.

Continuous Integration via Model Merging

For new corpora, a fresh MEMORY MODEL is trained per corpus D_i, all initialized from the same base M_phi_0. Task vectors tau_i = phi_i - phi_0 are merged using linear weighting or TIES merging:

phi_merged = Merge(phi_0, {tau_i})

TIES merging saves 33% compute at K=2 corpora and 5.5x at K=10, with measurable but acceptable accuracy loss versus full retraining.

Method Comparison

| Property | Non-parametric (RAG) | Parametric (SFT) | Implicit Memory | MEMO | |---|---|---|---|---| | Frozen base LLM | yes | no | yes | yes | | No retrieval index | no | yes | yes | yes | | Black-box compatible (closed models) | yes | no | no | yes | | No catastrophic forgetting | yes | no | yes | yes | | Constant-size memory | no | yes | no | yes | | Cross-model transferable | yes | no | no | yes |

MEMO is the only method satisfying all six properties.

Limitations

Reflective QA generation is expensive: ~240 GPU-hours, plus ~180 GPU-hours training, totaling ~420 GPU-hours per corpus — cheaper than full SFT but pricier than index construction.
Query-type coverage depends on the GENERATOR's imagination; out-of-distribution queries may degrade performance (a common limitation of parametric methods).
The lead over RAG on pure retrieval accuracy is narrowing as RAG systems improve; MEMO's structural advantages sit in cross-document reasoning and noise-robustness settings.
Even though executives can be black-box APIs, high-frequency calls can make API costs exceed local RAG.

Why It Matters

From appendage to module: memory becomes a standalone trainable, deployable, swappable unit, mirroring how GPUs evolved from CPU coprocessors to independent devices.
From retrieval to internalization: bottleneck shifts from retrieval-algorithm quality to data-synthesis quality, favoring long-context, multi-hop scenarios.
From coupling to transfer: one MEMORY MODEL can serve multiple EXECUTIVE MODELs. A single model, as small as 1.5B, can memorize an entire corpus, enabling personal knowledge bases that plug into various commercial assistants without vector databases or retrieval pipelines.

References

arXiv: 2605.15156 — Quek et al., "MEMO: Memory as a Model", May 2026
Marktechpost coverage: https://www.marktechpost.com/2026/05/26/memo-a-modular-framework/
Related: NeurIPS 2023 LongMem paper on decoupled memory