EngramEdit: Decoupled Knowledge Updates in LLMs via Conditional Memory

Forum topic · 小凯 · 2026-10-09

Summary

EngramEdit (arXiv 2610.10533) is a knowledge editing method built on conditional memory architectures such as DeepSeek Engram, which use input n-grams to look up learned embeddings and expand LLM capacity with minimal extra computation. The key challenge is that different expressions of a fact may activate different n-gram embeddings, and updating shared embeddings can unintentionally alter predictions about unrelated facts. EngramEdit addresses this by first computing target memory representations so the model predicts the updated fact across multiple phrasings, then jointly updating shared n-gram embeddings to match these targets while applying stronger penalties to frequently reused embeddings to protect unrelated knowledge. Experiments show near-perfect editing success rates, generalization to unseen phrasings and multi-hop reasoning (with chain-of-thought accuracy nearly three times that of the strongest baseline), and negligible impact on unrelated knowledge and general capabilities during cumulative fact updates.

EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

Field: NLP
Authors: Hongru Cai, Ran Wei, Wenjie Wang, Chengfa Wu, Ning Song, Yongqi Li, Wenjie Li
Published: 2026-10-07
arXiv: 2610.10533

Background

Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed.

The Challenge

Realizing this potential is difficult because different expressions of a fact may activate different n-gram embeddings, while updating shared embeddings can unintentionally change the model's predictions about other facts.

Method

EngramEdit performs decoupled knowledge updates through conditional memory in two stages:

1. Target computation: It first computes target memory representations so the model can predict the updated fact under multiple phrasings. 2. Joint embedding update: It then jointly updates shared n-gram embeddings to match these targets across phrasings and edits, applying stronger penalties to frequently reused embeddings to protect unrelated knowledge.

Results

Near-perfect editing success rates through conditional memory.
Updated knowledge generalizes to unseen phrasings and multi-hop reasoning, with chain-of-thought accuracy nearly three times that of the strongest baseline.
Unrelated knowledge and general capabilities remain largely unaffected across the accumulation of factual updates.