Key points
- Conditional memory paradigm: Engram separates static knowledge storage from dynamic neural computation, complementing MoE's conditional computation with a new sparsity dimension called conditional memory.
- O(1) knowledge lookup: Modernized hash N-gram embeddings with multi-head hashing reduce storage complexity from O(V^N) to O(V × k), enabling constant-time retrieval.
- Context-aware gating: An attention-based gate dynamically weights retrieved N-gram embeddings to handle polysemy and context-dependent meaning.
- U-shaped scaling law: Optimal allocation splits sparse capacity into ~75-80% MoE (computation) and ~20-25% Engram (memory), proving pure MoE architectures are suboptimal.
- Benchmark gains (iso-parameter, iso-FLOP): Engram-27B improves over pure MoE baselines by +3.4% on MMLU, +5.0% on BBH, +4.0% on CMMLU, and raises Multi-Query NIAH long-context retrieval from 84.2% to 97.0%.
- Infrastructure efficiency: Deterministic lookup indices enable asynchronous prefetching, allowing a 100B-parameter embedding table to be offloaded to CPU DRAM via PCIe.
- Knowledge-intensive tasks: MMLU +3.4%, CMMLU +4.0% under matched compute and parameters.
- Reasoning: BBH +5.0%, because Engram offloads shallow pattern reconstruction, effectively deepening the network for downstream reasoning.
- Long-context retrieval: Multi-Query NIAH jumps from 84.2% to 97.0% by separating local pattern lookup from global context modeling.
- Knowledge-intensive systems: medical QA, legal analysis, financial decision support, terminology processing.
- Reasoning-intensive systems: code generation, mathematical problem solving, scientific research assistance.
- Long-context systems: document summarization, multi-turn dialogue, complex task planning.
- Deployment benefit: Deterministic indices enable prefetching, reducing dependence on expensive GPU memory and offering an alternative to raw parameter or FLOP scaling.
- [1] https://conditionalmemory.org/
- [2] https://github.com/deepseek-ai/Engram
- [3] https://www.36kr.com/p/3637481791341824
- [4] https://eu.36kr.com/en/p/3637114445349889
- [5] https://finance.biggo.com/news/202601130451_DeepSeek-Engram-New-AI-Architecture-Launch
- [6] https://www.arxiv.org/pdf/2601.07372
- [7] https://www.c114.net.cn/industry/51609.html
- [8] https://github.com/deepseek-ai/Engram
- [9] https://pandaily.com/deep-seek-open-sources-engram-memory-module-introducing-a-new-dimension-for-ll-ms
- [10] https://eu.36kr.com/en/p/3637114445349889
- [11] https://www.tomshardware.com/tech-industry/artificial-intelligence/deepseek-touts-memory-breakthrough-engram
- [12] https://blog.csdn.net/youcans/article/details/156914214
- [13] https://arxiv.org/html/2601.07372v1
Architecture details
Core design philosophy
Transformer-based LLMs lack a native lookup mechanism for knowledge. They must "rebuild a static lookup table" through layered computation—for example, inferring that "Diana, Princess of Wales" refers to a specific person only after multiple layers process the constituent tokens. Engram addresses this by treating memory and computation as complementary subsystems rather than collapsing both into the same parameters.
Two-stage pipeline
1. Retrieval stage: hash(N-gram) → index → embedding performs O(1) lookup across multiple smaller embedding tables via independent hash functions.
2. Fusion stage: gate = attention(h, e_ngram) uses context-aware gating to blend retrieved static embeddings with the Transformer's hidden state, resolving one-to-many mappings.
MoE vs Engram comparison
| Dimension | MoE (conditional computation) | Engram (conditional memory) | |---|---|---| | Primary goal | Conditionally activate neural experts | Conditionally trigger static memory lookup | | Computation type | Dense neural computation | O(1) deterministic table lookup | | Typical placement | Deeper layers for complex reasoning | Early layers for pattern retrieval |