MeMo: Memory as a Model — Giving LLMs an External Hard Drive Without Touching Weights

> "Memory is the dust of the soul." — Marcel Proust

The Frozen Problem

LLMs are "frozen time capsules": after pretraining, their parameters are locked, so their knowledge stops at the training data cutoff. Existing approaches to injecting new knowledge all have drawbacks:

• Retraining: costs millions of dollars and months of time — impractical for frequently updated knowledge.
• Fine-tuning: cheaper but risks catastrophic forgetting of old knowledge.
• RAG: low cost, but retrieval quality is unstable and long contexts burden the model.
• In-context learning: cheap but limited by context length and must be re-supplied every time.

MeMo's Core Idea: Memory as a Model

From MIT CSAIL and Singapore-based researchers, MeMo's idea is surprisingly simple:

> Do not modify the LLM's parameters. Instead, pair it with an independent, trainable memory model dedicated to learning new knowledge.

The analogy: a learned professor (the LLM) with deep knowledge but poor recall of specific sources, paired with a librarian with a photographic memory (the memory model). When a question arrives, the librarian retrieves relevant material, digests and integrates it, then presents it in a format the professor can easily use.

Architecture:

• Frozen LLM: keeps all general knowledge and reasoning ability intact.
• Memory Model: a separate trainable model that learns, integrates, compresses, and structures new knowledge.

How MeMo Differs from RAG

RAG is a "porter" that moves retrieved snippets into the LLM's context. MeMo is a "translator" that reformulates knowledge for the LLM. The memory model:

1. Models cross-document relationships — it sees multiple documents during training and learns cross-references, causal links, and complementary information that isolated RAG snippets cannot capture. 2. Is robust to retrieval noise — via attention mechanisms, it filters irrelevant results instead of blindly passing them into the context. 3. Decouples retrieval cost from corpus size — knowledge is internalized, so inference-time cost does not grow with the size of the knowledge base.

Training Objective

The memory model is trained to produce an intermediate representation (a compact knowledge summary) that maximizes the frozen LLM's chance of answering correctly. Given a question plus relevant documents, the memory model generates a summary; the frozen LLM then answers from question + summary. The answer loss is backpropagated only into the memory model — the LLM's weights never change. It is essentially training a "prompt engineer" whose prompts are optimized end-to-end.

Experimental Results

MeMo was evaluated on three benchmarks:

• BrowseComp-Plus (agentic web browsing, multi-source extraction): significantly better accuracy than RAG variants in multi-hop reasoning scenarios.
• NarrativeQA (question answering over entire books): substantially outperforms simple retrieval; the memory model learns "plot compression" — distilling long narratives into key-event timelines.
• MuSiQue (multi-hop question chains): notably lower error rates; the memory model learns representations of reasoning chains rather than isolated facts.

Why It Works

1. Plug-in beats surgery: keeping the base model untouched avoids catastrophic forgetting entirely. 2. Specialization beats generalization: the LLM handles reasoning and language; the memory model handles knowledge integration. 3. Pluggability: per the paper, no access to LLM weights or output logits is needed — it works with open-source and closed-source models (e.g., GPT-4, Claude, Llama), and the memory model can be updated independently.

Limitations and Open Questions

• Memory capacity: the memory model itself is finite; behavior at internet-scale corpora is not explored.
• Training data dependency: requires question–document–answer pairs, which may be scarce in some domains.
• Forgetting within the memory model: whether it forgets old knowledge bases when learning new ones is not clearly addressed.
• Real-time knowledge: internalized knowledge requires retraining the memory model; traditional RAG may suit news or stock-price scenarios better.
• Multimodality: only text knowledge is discussed.

Outlook: Modular AI Brains

MeMo hints at a broader trend — modular AI systems with a general reasoning core plus pluggable modules for memory, tools, vision, specialized reasoning, and safety. Benefits include scalability, maintainability, interpretability, and safer isolation of sensitive functions.

Conclusion

MeMo is conceptually simple and practically useful: rather than making one model do everything, give it a dedicated partner for the one thing it cannot do. It gives LLMs extensible, updatable, non-forgetting memory — a solid step toward adaptive, lifelong-learning AI.

References

• Lee, S., Leong, A. W. L., Verma, A., Prakash, A., Chen, N. F., Low, B. K. H., Rus, D., & Solar-Lezama, A. (2026). *MeMo: Memory as a Model*. arXiv:2605.15156.
• Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Systems*, 33, 9459-9474.
• Proust, M. (1913). *Du côté de chez Swann* (Swann's Way). Vol. 1 of *À la recherche du temps perdu*.