MeMo: Memory as a Model — Teaching LLMs New Knowledge Without Touching Their Weights

> "Memory is the diary we all carry about with us." (The original post opens with a Proust quote on memory.)

The Problem: LLMs Are Frozen in Time

After pre-training, a large language model's parameters are effectively locked — it doesn't know yesterday's news, last week's papers, or the current date's facts. The industry has explored several workarounds:

• Retraining: mixing new knowledge into pre-training data. Costs millions of dollars and months of time — impractical for frequently updated knowledge.
• Fine-tuning: gradient updates on new data. Cheaper, but risks catastrophic forgetting, where learning new facts erases old ones.
• RAG (Retrieval-Augmented Generation): store knowledge externally and retrieve relevant chunks as context. Cheap, but retrieval quality is unstable, context windows get long, and retrieval cost scales with corpus size.
• In-context learning: stuff knowledge into the prompt. Extremely cheap, but limited by context length and must be re-entered every time.

MeMo's Core Idea: Memory as a Model

MeMo, proposed by researchers from MIT CSAIL and Singapore, keeps the LLM completely frozen and attaches an independent, trainable memory model. The analogy in the post: a learned but forgetful professor (the LLM) paired with a young librarian with a photographic memory (the memory model).

The memory model does not just fetch documents — it reads, digests, and integrates them, then presents a condensed representation in a format the frozen LLM can use effectively.

Why It's Not Just RAG

• Cross-document relationship modeling: trained over multiple documents simultaneously, the memory model learns cross-references, causal links, and complementary information that isolated RAG chunks cannot capture.
• Robustness to retrieval noise: via attention mechanisms it filters irrelevant results instead of passing them raw into the LLM's context.
• Retrieval cost decoupled from corpus size: knowledge is internalized rather than searched in an external index, so inference-time cost doesn't grow with the corpus.

Training Objective

The memory model is not trained to answer questions directly. Instead:

1. Input: a question plus relevant knowledge documents. 2. The memory model generates a compact "knowledge summary." 3. The frozen LLM receives question + summary and produces the final answer. 4. The answer loss backpropagates only into the memory model; LLM parameters stay fixed.

Effectively, it trains a "prompt engineer" whose job is to produce the representation that best helps the LLM answer correctly.

Benchmark Results

• BrowseComp-Plus (multi-page browsing and cross-source verification): significant accuracy gains in multi-hop reasoning over RAG variants.
• NarrativeQA (book-length narrative comprehension): substantially better long-context memory; the memory model learns "plot compression" into event timelines.
• MuSiQue (multi-hop chained questions): notably lower error rates; the model learns representations of reasoning chains rather than isolated facts.

Analysis: Why It Works

• "External attach" beats "surgery": no parameter changes means no catastrophic forgetting in the base model.
• Specialization over generalization: the LLM handles general reasoning; the memory model handles knowledge integration.
• Pluggability: per the paper, it needs no access to LLM weights or output logits — it works with open and closed models alike (GPT-4, Claude, Gemini, Llama). Enterprises without access to internal parameters can still inject domain knowledge.

Limitations and Open Questions

1. Capacity limits: the memory model itself cannot absorb unlimited knowledge; behavior at internet-scale corpora is undiscussed. 2. Training data dependency: it needs question–document–answer triples, which are scarce in some domains. 3. Forgetting within the memory model: whether it forgets old corpora when learning new ones is not addressed. 4. Real-time knowledge: internalized knowledge requires retraining the memory model; real-time RAG may suit news or stock prices better. 5. Multimodality: the paper covers text only.

Outlook: Modular AI Brains

The post situates MeMo in a broader trend toward modular AI: a general reasoning core plus pluggable add-ons — memory, tools, vision, specialized reasoning, and safety modules — each independently trained, updated, and replaceable. Benefits include scalability, maintainability, interpretability, and security isolation.

Conclusion

MeMo is conceptually simple and practically useful: rather than making the model more complex, give it a dedicated partner for memory. It grants LLMs "the ability to learn knowledge" without being "reborn" each time — a step toward adaptive, lifelong-learning AI.

References

• Lee, S., Leong, A. W. L., Verma, A., Prakash, A., Chen, N. F., Low, B. K. H., Rus, D., & Solar-Lezama, A. (2026). *MeMo: Memory as a Model*. arXiv:2605.15156.
• Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Systems*, 33, 9459-9474.
• Proust, M. (1913). *Du côté de chez Swann* (Swann's Way). Vol. 1 of *À la recherche du temps perdu* (In Search of Lost Time).