Paper Information
| Item | Detail | |---|---| | Title | Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory | | Authors | Runxi Cheng, Yuchen Guan, Yongxian Wei, Qianpu Sun, Qixiu Li, Sinan Du, Feng Xiong, Chun Yuan, Yan Lu, Yeyun Gong (10 authors) | | Institutions | Microsoft Research Asia, Tsinghua University | | arXiv ID | 2605.20948 | | Date | May 20, 2026 | | Category | cs.CL | | Size | 25 pages, 12 figures, 5 tables | | Core claim | Use a trained model as a "memory donor": freeze its hidden states into an offline conditional memory that a new model retrieves during pre-training—trading minimal training and inference overhead for a significant boost in effective model capacity |
---
Every child knows a lion is bigger than a cat. But ask a six-year-old *why* a lion is bigger than a cat, and they'll say "because a lion is a big cat." Ask why a tiger is also bigger, and after a pause: "because... a tiger is also a big cat."
The answer isn't wrong. But it doesn't reflect real understanding of species evolution, food chains, or ecological niches. The child is just doing analogy—comparing everything large and feline back to a cat.
Language models today are, in a sense, children. They consume the internet's text token by token from scratch, gradually building an understanding of the world. This is pre-training, and whatever the model "knows" ends up embedded in its parameters—tens or hundreds of billions of floating-point numbers.
But pre-training is extraordinarily expensive. At current trends, the cost of training a frontier model grows tenfold every two years—exponential, not linear. Compute is increasing, but costs grow faster.
Is there a way for a new model, during pre-training, to avoid starting from zero—not by fine-tuning an old model, but by borrowing ready-made knowledge and plugging it into its own architecture like Lego bricks?
That is what this paper addresses. Its solution has an aptly precise name: Memory Grafting.
---
🌱 Step One: Understanding Engram
Before Memory Grafting, there was Engram.
Engram's idea: a language model doesn't have to rely only on parameters. It can also carry an "external memory table"—a huge key-value store. Keys are n-grams of text (contiguous token sequences); values are the "hidden states" those n-grams induce—the internal activation vectors a model produces while reading that text.
Think of reading *The Three-Body Problem*. When you hit the word "droplet," your brain activates a cluster of related concepts—strong nuclear force, absolute zero, cosmic sociology. Those activated states, recorded as vectors, are your "Three-Body memory." Next time you see "droplet," you don't recompute; you just look it up.
Engram does exactly this. During pre-training it jointly learns two things: the normal Transformer parameters and an external memory table. On encountering an n-gram, it checks whether the table already stores a hidden state for it; if yes, reuse it; if no, compute and store a new one.
The problem is the "jointly" part. Because the memory table and the model parameters are trained in a coupled fashion—the table updates along with the model—the memory can't be scaled very large. Once the table grows, training overhead becomes unbearable. The Engram paper itself admits that scaling the table from 8M keys to 16M keys made the gains vanish.
In other words: you make the model both athlete and referee during pre-training—learning linguistic regularities while memorizing specific facts. The two compete for resources, and both end up underperforming.
---
🧬 The Essence of Grafting: Who Operates, Who Is Operated On
Memory Grafting does something simple but radical: separate "learning linguistic regularities" from "memorizing specific facts," and hand them to two different models.
It introduces two roles:
Donor model: an already pre-trained model. It requires no further training; its parameters are frozen. Its only job is, given an n-gram, to run a forward pass and emit the hidden state of its last token.
Recipient model: the model being trained from scratch. It must learn linguistic regularities—but when it encounters an n-gram, it doesn't compute the hidden state itself; it looks it up in the memory store the donor pre-built.
The procedure is minimal:
1. Collect all frequent n-grams in the training corpus (the paper uses 2-grams and 3-grams). 2. Feed each n-gram to the donor model and record the hidden-state vector of its final token. 3. Store these vectors in a key-value table: keys are the hashed n-grams, values the vectors. 4. During recipient training, for each token, check whether the preceding n-gram is in the table. If yes, retrieve the vector, pass it through a lightweight linear projection and a gate, and inject it into the computation. If no, fall back to a smaller, locally trained Engram table.
The operative keyword is offline. The donor works only in the first two steps—the memory table is built before recipient training begins. After that, the donor retires. At training time, the recipient performs a single O(1) table lookup.
That is what "grafting" means. It's not a child learning under a parent's constant supervision. It's parents writing an encyclopedia in advance and placing it on the child's desk, to be consulted whenever needed.
---
💰 Why This Matters: A Memory Table That's Essentially Free
The economics fit in one sentence: the memory store can be scaled arbitrarily while training and inference costs stay nearly constant.
The donor's hidden states are frozen. No backpropagation updates them. No GPU memory is needed for their gradients. The only cost is the one-time forward passes to build the table—and an O(1) hash lookup afterwards.
What does that mean? If you have a 35B-parameter donor (say, Qwen3.5-35B), you can compress its "text understanding" into a key-value table and transfer that capability to a 0.92B model—without increasing the small model's training time or inference latency, just by adding a table.
The experimental numbers are telling:
- 2.8B recipient: average benchmark score rises from 51.95 (MoE baseline) and 52.43 (vanilla Engram) to 53.86.
- 0.92B recipient: all donor variants beat the baselines, with Qwen3.5-35B-A3B as donor giving the strongest gains.
---
🔬 Methodology in the Details: Longest-Suffix Matching and Hash Fallback
Several engineering details deserve separate mention, because they determine whether the method works in practice.
Longest-suffix matching. Retrieval isn't a naive "look up the current 2-gram." The recipient searches for the *longest* suffix of the current text that matches any key. If the table contains both "droplet strikes" and "droplet strikes humanity," it prefers the latter—longer suffix, more context. Simple but crucial: it lets the recipient use the most relevant memory, not merely the most frequent. In long documents, this longer-is-more-precise strategy naturally prioritizes more specific knowledge.
Hash-based Engram fallback. Not every n-gram is in the donor's memory table. To cover misses, the paper keeps a smaller, locally trained Engram table with hash-based lookup. It's like mostly consulting the encyclopedia, occasionally flipping through your own pocket dictionary.
Lightweight projection and gating. Vectors retrieved from the donor can't be injected directly into the recipient's computation—the two models may differ in dimensionality and semantic space. A small linear projection layer acts as a "translator" from the donor's representation space to the recipient's; a gating mechanism acts as a "valve," deciding how much of each memory should flow into the current inference step.
All three components are described thoroughly and explicitly in the methodology—no black boxes.
---
📊 Experiments: Who Makes the Best Donor
The paper's systematic comparison is impressive: multiple donors, multiple recipients, multiple memory-table sizes, cross-validated.
Key findings:
Bigger donors work better. Qwen3.5-35B-A3B produced the largest gains on the 0.92B recipient—not a 7B donor, not a 14B one. The donor's knowledge density flows directly into the recipient's performance.
The recipient needs a minimum capacity. At 150M scale, Memory Grafting's gains were smaller than at 0.92B and 2.8B. The paper's explanation: an overly small recipient is too limited to fully "digest" even better external memory—like a five-year-old with an encyclopedia who still can't follow a quantum chromodynamics description of the strong force.
Comparison with MoE. Both approaches try to increase effective capacity at relatively low inference cost. The difference: MoE decides at training time which parameters activate, while Memory Grafting's external memory is frozen and offline. At 2.8B, Memory Grafting (53.86) beats MoE (51.95). That's an informative comparison—in some settings, borrowed knowledge beats activating more parameters.
Comparison with vanilla Engram. This is the core ablation. Vanilla Engram's gains vanish when the table grows from 8M to 16M keys—updating memory and parameters jointly creates gradient conflicts that worsen with table size. Memory Grafting decouples the two, so it supports much larger memory stores.
---
❓ What the Paper Doesn't Answer
Honest list of open questions:
Donor selection. Several donors were tested (e.g., Qwen3.5-35B-A3B), but no principled selection guide is given. Does an architecturally similar donor beat a cross-architecture one? Does a domain-matched donor beat a general-purpose one? All open.
Scaling limits of the memory table. What's the maximum table size studied? Where do marginal returns start declining? Lookup is O(1) in theory, but in real deployments hash collisions, memory bandwidth, and cache hit rates may become bottlenecks. No large-scale deployment profile is provided.
Corpus coupling. The donor is trained on some corpus—must the recipient's corpus overlap heavily? If the recipient targets a new domain (e.g., medicine) and grafts from a general-corpus donor, how well does that work? Cross-domain scenarios weren't tested.
Cognitive framing. Is storing one model's hidden states for another model's use compression, distillation, or something closer to neuroscientific memory transplantation? The paper doesn't explore this, but the discussion matters for the technology's long-term significance.
Safety. The donor's memory may encode specific text fragments—personal data, copyrighted content, or biased statements. Could these be transferred to the recipient via grafting? Not discussed.
---
🦾 Paradigm Shift: From "Train Bigger" to "Borrow Existing Intelligence"
Step back. Memory Grafting represents not just a new training trick but a new way of thinking.
For a decade, progress has obeyed an iron law: performance = f(model size, data size, compute). All three grow, performance rises. That's the Scaling Law.
The problem isn't accuracy—Scaling Laws are extremely accurate. The problem is the hidden assumption: every new model must re-experience the full pre-training journey from zero. Every time you train a GPT-5, you re-burn the entire evolutionary history from 0 to 175 billion parameters.
Memory Grafting suggests a different path: compositional pre-training. Instead of stacking parameters from scratch, treat trained models as "memory bricks" to be slotted into new models on demand. A donor only needs to run once (to build the table), then can be reused across unlimited recipients.
The implicit conclusion is bold: knowledge transfer between models can be orders of magnitude more efficient than training the model itself. Not distillation—distillation needs heavy compute to match output distributions. Not fine-tuning—fine-tuning only changes parameters within one model. This directly extracts a model's hidden states, freezes them, and transfers that knowledge via O(1) lookups.
Five years from now, Memory Grafting may not be the final design. But the direction is right: turn pre-training from heavy industry into light industry. From a steel mill into a Lego factory.
---
💭 Coda: Back to the Child and the Encyclopedia
Years ago, a cognitive scientist told me about a study. Two groups of children: the first memorized multiplication tables, then learned word problems; the second learned word problems directly, no tables.
The unsettling result: on day one, both groups performed equally. Two weeks later, the second group had forgotten most of it; the first group had barely forgotten anything.
The multiplication table isn't understanding. But it gives a child a mental scaffold that can be retrieved without recomputation—and understanding grows on top of the scaffold.
Language model pre-training resembles the second group: every new word, every new context, requires re-deriving regularities from raw data. Deep, perhaps—but extremely inefficient.
Memory Grafting hands the model a multiplication table. A cheat sheet someone else already organized, that it never had to memorize from scratch. Not a replacement for understanding—a way to reach the starting line of understanding faster.
The paper's conclusion is one precise sentence: "Pretrained models can serve as reusable constructors of external latent memory."
Three words: reusable. Constructive. Transferable.
More honest than any benchmark number in the paper.
---
📚 References
1. Cheng, R., Guan, Y., Wei, Y., Sun, Q., Li, Q., Du, S., Xiong, F., Yuan, C., Lu, Y., & Gong, Y. (2026). Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory. *arXiv:2605.20948*. 2. Chen, Y., Wang, J., & Chen, T. (2025). Engram: Scaling Conditional Memory for Language Models. *arXiv*. 3. Shazeer, N., Mirhoseini, A., Maziarz, K., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. *ICLR*. 4. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531. 5. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.