DeepSeek Engram: 75% Thinking + 25% Memory Is the Optimal Split
DeepSeek recently published a paper titled *Conditional Memory via Scalable Lookup*, arguing that today's large models waste compute simulating something cheap: looking up a dictionary. They propose a module called Engram to give models true memory, and through experiments compute an optimal ratio: 75% thinking + 25% memory.
1. The Problem: LLMs Waste Compute on "Memorization"
When a model sees "Diana, Princess of Wales", cited research shows it needs multiple layers of attention and FFN to progressively identify the entity:
| Layer | Internal representation | |---|---| | 1-2 | "Wales" (treated as a UK region) | | 3 | "A European country" | | 4 | "Title held by female monarchs" | | 5 | "Wife of the Prince of Wales" | | 6 | "Princess Diana (1961-1997), ex-wife of Prince Charles" |
It took six full layers just to identify a fixed historical entity. This information is static — recomputing it every time is like deriving "the capital of China" from first principles instead of opening a dictionary. Models spend substantial network depth on repeatedly rebuilding static knowledge.
2. Engram's Core Idea: Give the Model a Dictionary
"Engram" comes from neuroscience, referring to the physical traces of memory in the brain. The mechanism:
1. Hash lookup: Phrases like "Alexander the Great" act as dictionary keys; hashing locates the entry instantly. 2. Direct retrieval: Pre-stored information is fetched directly — flipping a page instead of thinking. 3. Context gating: Handles ambiguity. For "apple", if the context is fruit, the model ignores the "Apple Inc." entry; if it's about phones, it adopts it.
Like a smart student who brings a dictionary but checks whether the entry matches the question before using it.
3. Key Finding: A U-Shaped Scaling Law
With a fixed total parameter count, DeepSeek studied how to split parameters between MoE (thinking) and Engram (memory), defining ratio ρ where ρ = 100% is pure MoE:
- Low ρ (pure Engram): Poor results — infinite memory but no reasoning, like Borges' Funes with perfect recall but no ability to think.
- High ρ (pure MoE): Suboptimal — strong reasoning but wasted compute rebuilding static knowledge.
- Optimum: At ρ ≈ 75%-80%, i.e. ~20-25% Engram + 75-80% MoE, validation loss is lowest.
- Knowledge: MMLU +3.4, CMMLU +4.0, MMLU-Pro +1.8
- Reasoning: BBH +5.0, ARC-Challenge +3.7, DROP +3.3
- Code & math: HumanEval +3.0, MATH +2.4, GSM8K +2.2
- Long context: In multi-question needle-in-a-haystack tests, accuracy rose from 84.2% to 97.0% — attention can focus on "what happened" once Engram handles "who is this".
- Engineering: Unlike MoE routing (decided at inference time, with unpredictable latency), lookup keys are known in advance. This enables asynchronous prefetching (prepare dictionary pages while layer 1 computes) and memory tiering (store the dictionary in cheap, large CPU RAM instead of GPU VRAM). A model with a 100-billion-parameter dictionary in CPU memory was only ~2% slower than the baseline.
Memory and thinking are complementary yet opposed: perfect memory kills thinking; pure thinking requires constant repetitive labor.
4. Experiments: Reasoning Gains Exceed Knowledge Gains
Surprisingly, Engram improved reasoning more than pure knowledge tasks:
Why: Layer-wise analysis shows models without Engram spend the first 5-6 layers identifying entities before reasoning begins. With Engram, entity information is available at layer 2, leaving 20+ layers for actual reasoning. Layer 5 of the Engram model has "thinking depth" equivalent to layer 12 of a normal model — effectively 7 free layers of depth.
5. Long-Context Gains and System Design
6. Conclusion
O(1) lookup always beats O(n) computation. Assign tasks to the right modules:
> Static knowledge → table lookup > Dynamic reasoning → computation
You don't re-derive 1+1 every time; you recall "2" from memory and spend the saved capacity on harder problems. DeepSeek's computed optimum — 75% thinking + 25% memory — may be a necessary step toward higher-order intelligence.