DeepSeek Engram: The Optimal 75% Reasoning + 25% Memory Split for LLMs

Forum topic · ✨步子哥 · 2026-01-14

Summary

DeepSeek's new paper, "Conditional Memory via Scalable Lookup," introduces Engram, a conditional memory module that gives large language models a fast lookup "dictionary" for static knowledge instead of wasting layers of computation to reconstruct fixed facts like named entities. The paper's key insight is a U-shaped scaling law: with a fixed parameter budget, models perform best when roughly 75-80% of parameters go to MoE (reasoning) and 20-25% to Engram (memory). Benchmarks show Engram improves reasoning even more than knowledge tasks (BBH +5.0, MMLU +3.4), because entity recognition moves to layer 2, freeing deeper layers for inference. Engram also boosts multi-question needle-in-a-haystack accuracy from 84.2% to 97.0%, and its hash-lookup design allows the dictionary to be offloaded to cheap CPU memory asynchronously, costing only ~2% speed with a 100B-parameter memory. The result suggests an optimal balance of 75% thinking plus 25% memory for future sparse architectures.

DeepSeek Engram: 75% Reasoning + 25% Memory

DeepSeek recently published a new paper titled *Conditional Memory via Scalable Lookup* with a striking core argument: current large language models waste enormous compute simulating a cheap operation—using expensive computation to look up a dictionary. They propose a module called Engram that gives models genuine "memory," and through experiments they computed an optimal ratio of memory to reasoning: 75% thinking + 25% memory.

1. The Problem: LLMs Waste Compute on "Rote Memorization"

What happens inside a model when it sees "Diana, Princess of Wales"? Research cited in the paper shows the model must consume multiple layers of Attention and FFN to progressively identify the entity:

| Layer | Emergent representation | |---|---| | 1–2 | "Wales" (treated as a region of Britain) | | 3 | "A country in Europe" | | 4 | "A title held by female monarchs" (begins recognizing Princess) | | 5 | "Wife of the Prince of Wales" | | 6 | "Princess Diana (1961–1997), ex-wife of Prince Charles" |

The model uses a full 6 layers just to identify a fixed historical figure. Yet this information is static—"Alexander the Great" is simply "Alexander the Great." Recomputing these fixed phrases and named entities through a neural network every time is like re-deriving the capital of China from first principles instead of opening a dictionary. LLMs squander network depth rebuilding static knowledge repeatedly.

2. Engram's Core Idea: Hand the Model a Dictionary

The name "Engram" comes from neuroscience—the physical structure in the brain that stores memory traces. The core idea is simple: give the model a "dictionary." Previously everything had to be worked out in the model's head; now it can bring a dictionary into the exam room. The mechanism:

1. Hash lookup: When the model sees words like "Alexander the Great" together, they serve as a dictionary index. A hash quickly locates the corresponding entry. 2. Direct retrieval: Pre-stored information is pulled out directly—no "thinking," just "page flipping." 3. Contextual gating: This resolves ambiguity. For "Apple" (fruit vs. the tech giant), the model checks context: if discussing fruit, it ignores the Apple Inc. entry; if discussing phones, it adopts it.

It's like a smart student who brings a dictionary but doesn't copy blindly—first checking whether the answer matches the question.

3. Key Finding: A U-Shaped Scaling Law

The paper's most insightful result concerns parameter allocation. DeepSeek studied how to split a fixed parameter budget between MoE (reasoning/computation) and Engram (memory/lookup), defining allocation ratio $\rho$ where $\rho = 100\%$ is pure MoE and $\rho < 100\%$ shifts some parameters to Engram. Results show a clear U shape:

Left end (low $\rho$ , pure Engram): Poor performance. Unlimited memory but no reasoning—like Borges' Funes, who remembered everything but could not think.
Right end (high $\rho$ , pure MoE): Suboptimal. Strong reasoning, but compute wasted on reconstructing static knowledge.
Bottom (optimum): At $\rho \approx 75\%-80\%$ , i.e., about 20–25% to Engram and 75–80% to MoE, validation loss is lowest.

This confirms a cognitive-science thesis: memory and reasoning are complementary yet opposed. Perfect memory kills thinking; pure thinking requires constant repetitive labor. DeepSeek's answer: 75% computation (thinking) + 25% memory.

4. Experimental Results: Reasoning Gains Exceed Knowledge Gains

Surprisingly, Engram—a "memory module"—improves reasoning benchmarks even more than pure knowledge tasks:

Knowledge tasks: MMLU: +3.4, CMMLU: +4.0, MMLU-Pro: +1.8

Reasoning tasks: BBH: +5.0, ARC-Challenge: +3.7, DROP: +3.3

Code & math: HumanEval: +3.0, MATH: +2.4, GSM8K: +2.2

Why memory boosts reasoning: Analyzing intermediate layer states, DeepSeek found that without Engram, the first 5–6 layers are busy identifying entities, and real reasoning only begins in much deeper layers. With Engram, entity information is available at layer 2, leaving the remaining 20+ layers fully for reasoning. Layer 5 of the Engram model has a "thinking depth" equivalent to layer 12 of a normal model—effectively a free 7-layer gain, freeing compute from memorization for problem-solving.

5. Long-Context Gains and System Design Innovations

Long context: Engram dramatically improves long-text handling. Multi-question needle-in-a-haystack accuracy rose from 84.2% to 97.0%. Because Engram answers "who is this," Attention can focus on "what happened," reducing cognitive load.

The dictionary can live in a drawer (system design): MoE models must decide on the fly which experts to route to, with unpredictable latency. But dictionary lookups are known in advance (seeing "Alexander" means looking up entries starting with A). This enables:

Asynchronous loading: While layer 1 computes, the system can prepare the dictionary pages needed for layer 2.
Memory offloading: The dictionary need not sit in expensive GPU VRAM (the "desktop")—it can live in cheap, large CPU memory (the "drawer").

Experiments show a model with a 100-billion-parameter dictionary in CPU memory runs only ~2% slower than one without, while gaining a huge knowledge store—meaning Engram can be scaled very large.

6. Conclusion: Balancing Memory and Thinking

The Engram paper not only proposes conditional memory as a foundational component for next-generation sparse models, but also reveals the essence of optimization at a cognitive level: O(1) table lookup will always beat O(n) computation.

Bigger and deeper is not always better—the key is assigning the right task to the right module:

> Static knowledge → lookup > Dynamic reasoning → compute

Like the human brain: you don't re-derive "1+1" every time—you recall "2" from memory and spend the saved brainpower on harder problems. DeepSeek's computed optimum—75% thinking + 25% memory—may well be a necessary step toward higher-order intelligence.