Gated DeltaNet-2: NVIDIA Decouples Erase and Write Gates in Linear Attention
TL;DR
NVIDIA (Ali Hatamizadeh, Yejin Choi, Jan Kautz) published Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (arXiv:2605.22791). The paper identifies a core bottleneck in the linear attention family: the traditional delta rule uses a single scalar to simultaneously control "erase old content" and "write new content" in the fixed-size recurrent state. Gated DeltaNet-2 splits this into two independent channel-wise gates (erase gate b_t and write gate w_t). At 1.3B parameters trained on 100B FineWeb-Edu tokens, it outperforms Mamba-2, Gated DeltaNet, KDA (Kimi Delta Attention), and Mamba-3 variants on language modeling, commonsense reasoning, and long-context retrieval — with the largest gains in multi-key needle-in-a-haystack scenarios.
| Attribute | Info | |---|---| | Paper | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | | Authors | Ali Hatamizadeh, Yejin Choi, Jan Kautz (NVIDIA) | | arXiv | 2605.22791 | | Core idea | Decouple erase gate and write gate, removing the scalar-binding constraint of the delta rule | | Scale | 1.3B params, 100B FineWeb-Edu tokens | | Training | AdamW, peak LR 4e-4, cosine decay, 1B warmup, global batch 0.5M | | Code | https://github.com/nvida/GatedDeltaNet2 (as cited in the paper) |
Background: the memory-editing bottleneck
Transformers give every token direct access to all history at quadratic cost. Linear attention replaces the unbounded KV cache with a fixed-size recurrent state, achieving linear-time sequence mixing and constant-memory decoding — but the state is a *compressed* key-value memory, so precise retrieval over long sequences becomes hard.
Evolution of state-control mechanisms:
- Mamba-2: data-dependent scalar decay; passive forgetting, no active editing.
- DeltaNet: delta rule — subtract the current read before writing, enabling precise overwrite; but scalar step size β_t controls both erase and write.
- Gated DeltaNet: delta rule + learnable decay gate; both gates scalar.
- KDA (Kimi Delta Attention): channel-wise decay on keys, but β_t still binds erase and write.
- Mamba-3: SSM route (exp-trapezoidal discretization, complex-valued transitions, MIMO).
- Gated DeltaNet-2 (this paper): channel-wise erase gate b_t + channel-wise write gate w_t.
- Erase gate \(b_t \in [0,1]^{d_k}\) (key-side, channel-wise)
- Write gate \(w_t \in [0,1]^{d_v}\) (value-side, channel-wise)
- The post notes that Alibaba's Qwen3.5 multimodal architecture reportedly adopts a hybrid Gated DeltaNet linear attention + full attention design, indicating industrial adoption of this line.
- KDA is from Moonshot AI (Kimi team); both NVIDIA and Moonshot independently advancing the "delta rule + gating" route suggests it is becoming the mainstream linear attention paradigm.
- NVIDIA's dense output around Mamba-2/linear attention aligns with its interest in constant-memory, high-throughput inference.
- Hatamizadeh, A., Choi, Y. & Kautz, J. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. *arXiv preprint* arXiv:2605.22791 (2026).
- Yang, S. et al. Gated DeltaNet. arXiv:2412.06446 (2024).
- Yang, S. et al. Kimi Delta Attention. arXiv:2502.09997 (2025).
- Gu, A. & Dao, T. Mamba. arXiv:2312.00752 (2023).
- Related zhichai discussion: https://zhichai.net/t/177619570
The core problem: one scalar controlling two things
Gated DeltaNet / KDA update rule:
The scalar β_t simultaneously controls: 1. Erase: the factor \((I - \beta_t k_t k_t^\top)\) determines how much old content is removed along the read direction. 2. Write: the term \(\beta_t k_t v_t^\top\) determines how much new content is committed.
The paper argues this is a modeling constraint, not a mathematical requirement: erasure is a key-side operation, writing is a value-side operation; tying them assumes "how much I want to forget" always equals "how much I want to learn."
Gated Delta Rule-2
Split β_t into:
With \(e_t = b_t \odot k_t\) and \(z_t = w_t \odot v_t\):
Note the asymmetry: the erase factor keeps \(k_t\) on the left (write direction) while the right factor \(b_t \odot k_t\) gives channel-wise read/erase selection.
Degenerate compatibility: with \(b_t = \beta_t \mathbf{1}_{d_k}\) and \(w_t = \beta_t \mathbf{1}_{d_v}\) the model reduces to KDA; with scalar decay as well, to Gated DeltaNet. The new model is a strict superset of prior work.
Efficient training: the WY recurrence survives
The paper shows cumulative channel-wise decay can be absorbed into both sides of the rank-one erase factor: with decay-normalized states \(\hat{S}_r = \text{Diag}(\gamma_r)^{-1} S_r\), the recursion becomes a purely asymmetric delta update, and the familiar WY formulation goes through with \(Y = A\bar{E}\), \(U = AZ\), where \(A = (I + \text{tril}(\bar{E}\bar{K}^\top, -1))^{-1}\) via triangular solves. Formula shapes match KDA exactly, so existing Triton kernels are largely reused. The only required backprop change is gate-aware gradients: channel-wise gates must stay inside the accumulated dot products (e.g., \(dA \mathrel{+}= dU \cdot Z^\top\) with \(Z = W \odot V\)).
Results
Language modeling (1.3B, 100B tokens, 4K context): Gated DeltaNet-2 achieves the best recurrent average (avg 48.4 across LAMBADA/PIQA/ARC/HellaSwag/WinoGrande etc.; WikiText PPL 10.28), beating KDA (47.8), Mamba-3 MIMO (47.6), Gated DeltaNet (47.3), and Mamba-2 (45.1).
RULER needle-in-a-haystack: largest gains in the interference-heavy settings:
| Model | S-NIAH-2 (4K) | S-NIAH-3 (4K) | MK-NIAH-1 (4K) | MK-NIAH-1 (8K) | |---|---|---|---|---| | Mamba-2 | 74.6 | 38.6 | 78.2 | 56.0 | | KDA | 90.8 | 64.2 | 92.6 | 78.8 | | Gated DeltaNet-2 | 93.2 | 69.8 | 95.2 | 83.6 |
Multi-key competing retrieval (MK-NIAH) is exactly where decoupled editing helps most, and performance degrades least from 4K to 8K context.
Ablations: channel-wise erase gate alone (scalar write gate) retains most of the gain (PPL 10.35, MK-NIAH 93.6); channel-wise write gate alone helps less (10.42 / 91.2); both scalarized drops to 10.48 / 89.4. The erase gate contributes most — selectively forgetting competing associations matters more than selectively writing. Extending the erase range to [0,2] brought no consistent gain.
Throughput: on H100, Gated DeltaNet-2 nearly matches KDA (38.0K → 36.1K tok/s across sequence lengths), a flat curve versus Transformer scaling.
Context and ecosystem signals
Limitations raised in the post
1. Scale: 1.3B is small; no 7B/13B/70B results. Gains may dilute at larger widths. 2. Hybrid attention ratio: optimal proportion of sliding-window attention layers in hybrids is not deeply explored. 3. Complementarity with Mamba-3: combining complex-valued SSM transitions with decoupled delta editing is unexplored. 4. Erase range: widening the erase gate to [0,2] showed no consistent benefit at this scale.