Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
TL;DR
NVIDIA published Gated DeltaNet-2 on arXiv, targeting the core bottleneck of the linear-attention family: traditional delta-rule updates use a single scalar to control both "erasing old content" and "writing new content," which causes unnecessary interference inside a fixed-size recurrent state. The paper proposes splitting these into two independent channel-wise gates — an erase gate \(b_t\) and a write gate \(w_t\). At a 1.3B-parameter / 100B-FineWeb-Edu training scale, the model outperforms Mamba-2, Gated DeltaNet, KDA (Kimi Delta Attention), and Mamba-3 variants on language modeling, common-sense reasoning, and long-context retrieval. The largest gains appear in RULER needle-in-a-haystack tasks under multi-key competition, and retrieval quality remains stable as context length grows.
---
Paper at a Glance
| Attribute | Detail | |---|---| | Paper | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | | Authors | Ali Hatamizadeh, Yejin Choi, Jan Kautz (NVIDIA) | | Date | 2026-05-21 | | arXiv | 2605.22791 | | Core idea | Decouple erase and write gates; remove the tied-scalar constraint | | Training | 1.3B params, 100B FineWeb-Edu tokens | | Setup | AdamW, peak LR 4e-4, cosine decay, 1B warmup, global batch 0.5M | | Code | https://github.com/nvida/GatedDeltaNet2 (as referenced) |
---
Background: The Memory-Editing Dilemma in Linear Attention
Linear attention replaces the unbounded Transformer KV cache with a fixed-size recurrent state, dropping sequence mixing to linear time and decoding memory to constant. The trade-off is direct: the state is a compressed key-value memory, so long sequences force many associations into a small space and exact retrieval becomes harder.
Evolution of Memory Control
| Method | Core Mechanism | Limitation | |---|---|---| | Mamba-2 | Data-dependent scalar decay α_t, global forgetting | No active editing; old content can only decay passively | | DeltaNet | Delta rule: subtract current read before writing; precise overwrite | Scalar step β_t controls both erase and write | | Gated DeltaNet | Delta rule + learnable decay gate | Both gates are scalars | | KDA (Kimi Delta Attention) | Channel-wise decay α_t (key dim), scalar β_t retained | β_t still controls both erase and write | | Mamba-3 | Exponential-trapezoidal discretization, complex state transitions, MIMO form | SSM path; does not subtract the current read | | Gated DeltaNet-2 (this paper) | Channel-wise erase gate \(b_t\) + channel-wise write gate \(w_t\) | — |
> Note: KDA is a Moonshot AI (Kimi team) work. NVIDIA explicitly lists it as a baseline, making this a direct comparison between two major labs on the same technical track.
---
The Core Problem: One Scalar for Two Different Things
The Tied-Scalar Expression
Recall the Gated DeltaNet / KDA update rule:
The scalar β_t simultaneously controls: 1. Erase: the left factor \((I - \beta_t k_t k_t^\top)\) decides how much old content is erased along the read direction. 2. Write: the right term \(\beta_t k_t v_t^\top\) decides how much new content is committed.
The paper argues that this is a modeling constraint, not a mathematical requirement. Erase is a key-side operation (which coordinates to remove from old reads); write is a value-side operation (which coordinates to commit). Using one scalar assumes "how much I want to erase" and "how much I want to write" are always aligned — which is plainly wrong in many situations.
---
Gated Delta Rule-2: The Decoupled Formulation
New Update Rule
Split the scalar into:
- Erase gate \(b_t \in [0,1]^{d_k}\): channel-wise on the key dimension.
- Write gate \(w_t \in [0,1]^{d_v}\): channel-wise on the value dimension.
- \(e_t = b_t \odot k_t\) (gated erase direction)
- \(z_t = w_t \odot v_t\) (gated write target)
- Key-side (erase): independently choose "from which key coordinates to read and erase old content."
- Value-side (write): independently choose "which value coordinates to commit into memory."
- \(b_t = \beta_t \mathbf{1}_{d_k}\) and \(w_t = \beta_t \mathbf{1}_{d_v}\) → collapses to KDA.
- Further \(\alpha_t = \alpha_t \mathbf{1}_{d_k}\) → collapses to Gated DeltaNet.
- \(Y = A \bar{E}\) (erase-side auxiliary)
- \(U = A Z\) (write-side auxiliary)
- Intra-chunk: \(O[n] = Q_\gamma S[n] + A_{qk}(U - YS[n])\)
- Inter-chunk state: \(S[n+1] = \text{Diag}(\gamma_C) S[n] + K_{\text{tail}}^\top (U - YS[n])\)
- \(dA \mathrel{+}= dU \cdot Z^\top\), where \(Z = W \odot V\)
- \(dA \mathrel{+}= dY \cdot \bar{E}^\top\), where \(\bar{E} = \gamma \odot (B \odot K)\)
- S-NIAH = Single Needle-In-A-Haystack
- MK-NIAH = Multi-Key Needle-In-A-Haystack
- Largest gains appear in the most interference-heavy scenarios (S-NIAH-2, S-NIAH-3, multi-key).
- Performance degrades least as context grows from 4K to 8K.
- MK-NIAH best showcases the value of decoupled editing: a fixed state must separate competing associations, which is exactly what independent erase/write control addresses.
- KDA: decay upgraded from scalar to channel-wise (key dim).
- Gated DeltaNet-2: delta gate further upgraded from scalar to dual channel-wise (key + value dims).
- Gated DeltaNet-2 collapses to KDA, indicating a natural continuation of the same line.
- Hatamizadeh, A., Choi, Y. & Kautz, J. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. *arXiv preprint* arXiv:2605.22791 (2026).
- Yang, S. et al. Gated DeltaNet: Improving Mamba-2 with Delta Rule. *arXiv preprint* arXiv:2412.06446 (2024).
- Yang, S. et al. Kimi Delta Attention: Native Sparse Attention with Delta Rule. *arXiv preprint* arXiv:2502.09997 (2025).
- Gu, A. & Dao, T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. *arXiv preprint* arXiv:2312.00752 (2023).
- Dao, T. & Gu, A. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. *ICML* (2024).
- Beck, M. et al. xLSTM: Extended Long Short-Term Memory. *NeurIPS* (2024).
- 智柴 discussion: 长上下文模型的「不可能三角」 — https://zhichai.net/t/177619570
Define the gated vectors:
The new Gated Delta Rule-2 update:
Expanded:
Key Insight: Asymmetric Erase Factor
Note the asymmetry: the erase matrix's left factor remains \(k_t\) (preserving the write direction), while its right factor becomes \(b_t \odot k_t\) (letting the read direction be channel-selected.
Backward Compatibility
The new model is a strict superset: it only adds degrees of freedom outside the tied-scalar subspace.
---
Efficient Training: The WY Representation Still Holds
Absorption Trick for Channel-Wise Decay
Does decoupling break chunkwise parallel training? No. The trick is to absorb accumulated channel-wise decay into both sides of the rank-one erase factor.
Define the decay-normalized state \(\hat{S}_r = \text{Diag}(\gamma_r)^{-1} S_r\), where \(\gamma_r\) is the accumulated decay. Substituting yields a pure asymmetric delta recurrence:
with \(\bar{k}_r = \gamma_r^{-1} \odot k_r\) and \(\bar{e}_r = \gamma_r \odot (b_r \odot k_r)\).
Channel-wise decay disappears — it is fully absorbed into the two sides of the erase factor.
WY Form
This lets us construct the standard WY auxiliary matrices:
with \(A = (I + \text{tril}(\bar{E}\bar{K}^\top, -1))^{-1}\) solved via triangular forward substitution.
Chunk outputs:
The formulas have the same shape as KDA; only how Y and U are built changes. Erase gates enter through \(\bar{E}\); write gates enter through \(Z\). Triangular solve + dense matmul — existing Triton kernels are reused.
Gate-Aware Backward Pass
With scalars, β_r could be pulled outside the cumulative-product as a shortcut. With channel-wise gates, that shortcut breaks: each gate factor must stay inside the accumulation.
Gradient computation:
This is the only mathematical change required to train Gated Delta Rule-2. All other backward kernels keep the same matrix shapes as KDA.
---
Experimental Results
Language Modeling and Common-Sense Reasoning
1.3B params, 100B FineWeb-Edu tokens, 4K context.
| Model | Type | WikiText PPL | LAMBADA Acc | PIQA | ARC-e | ARC-c | OpenBookQA | BoolQ | HellaSwag | WinoGrande | Avg | |---|---|---|---|---|---|---|---|---|---|---|---| | Mamba-2 | Recurrent | 11.34 | 32.3 | 72.0 | 52.8 | 25.4 | 32.4 | 55.4 | 34.2 | 55.6 | 45.1 | | DeltaNet | Recurrent | 10.81 | 34.6 | 73.0 | 54.6 | 26.1 | 32.6 | 55.8 | 35.4 | 56.1 | 46.5 | | Gated DeltaNet | Recurrent | 10.56 | 36.2 | 73.5 | 55.4 | 26.8 | 33.2 | 56.1 | 36.2 | 56.9 | 47.3 | | KDA | Recurrent | 10.41 | 37.8 | 74.1 | 56.0 | 27.2 | 33.8 | 56.5 | 37.0 | 57.4 | 47.8 | | Gated DeltaNet-2 | Recurrent | 10.28 | 39.1 | 74.6 | 56.8 | 27.8 | 34.4 | 57.0 | 37.8 | 58.0 | 48.4 | | Mamba-3 (SISO) | Recurrent | 10.52 | 36.5 | 73.2 | 55.2 | 26.5 | 33.0 | 56.0 | 36.0 | 56.5 | 47.1 | | Mamba-3 (MIMO) | Recurrent | 10.35 | 38.0 | 73.9 | 55.8 | 27.0 | 33.6 | 56.4 | 37.2 | 57.2 | 47.6 |
Gated DeltaNet-2 achieves the best average under the recurrent setting. Since state sizes match, the gain points to a stronger update rule rather than larger memory capacity.
RULER Needle-in-a-Haystack (Long-Context Retrieval)
| Model | S-NIAH-1 (4K) | S-NIAH-2 (4K) | S-NIAH-3 (4K) | S-NIAH-1 (8K) | S-NIAH-2 (8K) | S-NIAH-3 (8K) | MK-NIAH-1 (4K) | MK-NIAH-1 (8K) | |---|---|---|---|---|---|---|---|---| | Mamba-2 | 100.0 | 74.6 | 38.6 | 98.8 | 60.8 | 25.4 | 78.2 | 56.0 | | DeltaNet | 100.0 | 85.2 | 52.4 | 99.4 | 72.6 | 38.2 | 86.4 | 68.2 | | Gated DeltaNet | 100.0 | 88.4 | 58.6 | 99.8 | 78.4 | 44.6 | 90.2 | 74.4 | | KDA | 100.0 | 90.8 | 64.2 | 100.0 | 82.6 | 50.8 | 92.6 | 78.8 | | Gated DeltaNet-2 | 100.0 | 93.2 | 69.8 | 100.0 | 87.4 | 56.2 | 95.2 | 83.6 | | Mamba-3 (MIMO) | 100.0 | 89.6 | 62.4 | 100.0 | 80.2 | 48.4 | 91.8 | 76.4 |
Key observations:
Real-World Retrieval
On 2K-truncated real-world tasks (SQuAD, TriviaQA, HotpotQA, Natural Questions, DROP), Gated DeltaNet-2 leads in both Recurrent and Hybrid settings, with the strongest edge on noisy-association recovery — a direct fit for selective erase and gated write.
Ablation: Both Gates Matter, Erase Gate More
| Configuration | WikiText PPL | LAMBADA Acc | S-NIAH-2 (4K) | MK-NIAH-1 (4K) | |---|---|---|---|---| | Full Gated DeltaNet-2 | 10.28 | 39.1 | 93.2 | 95.2 | | Channel-wise \(b_t\) only (scalar \(w_t\)) | 10.35 | 38.4 | 91.8 | 93.6 | | Channel-wise \(w_t\) only (scalar \(b_t\)) | 10.42 | 37.6 | 89.4 | 91.2 | | Both gates scalar | 10.48 | 37.0 | 87.6 | 89.4 |
The erase gate contributes most of the gain. In multi-association competition, "selectively forget which old content" matters more than "selectively write which new content."
Training Throughput
On H100, Gated DeltaNet-2 tracks KDA's near-flat sequence-length scaling curve (38.0K → 36.1K tok/s), versus Transformer's steep drop. Overhead from channel-wise gating is minimal.
---
Ecosystem Signals
Qwen3.5 Has Already Adopted Gated DeltaNet
Alibaba's Qwen3.5 multimodal architecture already uses a "hybrid Gated DeltaNet linear attention + global attention" design, with 397B-A17B MoE and 27B dense variants. This means Gated DeltaNet is already an industry-adopted path, and Gated DeltaNet-2's migration path is clear.
NVIDIA's Strategy
All authors are from NVIDIA. The lab's sustained output on Mamba-2 and linear attention suggests an active effort to build a Transformer-replacement ecosystem — aligned with NVIDIA's commercial interest in inference optimization, since linear attention offers constant-memory decoding that benefits edge deployment and high-throughput serving.
Relationship to KDA
Two major labs (NVIDIA + Moonshot AI) advancing the same delta-rule + gating track suggests it is becoming the mainstream paradigm for linear attention.
---
Limitations and Open Questions
1. Scale validation: 1.3B is small. Whether gains hold or get diluted at 7B / 13B / 70B remains untested. 2. Hybrid attention ratio: optimal mix with sliding-window attention is not explored. Does pure recurrent suffice? 3. Complementarity with Mamba-3: Mamba-3 takes an SSM path with complex state transitions; combining it with decoupled editing is unexplored but theoretically interesting. 4. Erase range: extending the erase gate from [0,1] to [0,2] (allowing "over-erasure") gives no consistent gain at this scale. Whether stronger erasure is needed at larger scales is open.
---