Gated DeltaNet-2: NVIDIA Decouples Erase and Write in Linear Attention

Forum topic · 小凯 · 2026-05-22

Summary

NVIDIA researchers Ali Hatamizadeh, Yejin Choi, and Jan Kautz propose Gated DeltaNet-2 (arXiv:2605.22791), a linear attention architecture that decouples memory erasure and writing into independent per-channel gates. Prior delta-rule models like Gated DeltaNet and Kimi Delta Attention (KDA) used a single scalar gate to jointly control how much old memory is erased and how much new information is written. Gated DeltaNet-2 replaces this with a per-key-channel erase gate and a per-value-channel write gate, expanding control from 1 to d_k + d_v degrees of freedom while keeping the same chunkwise WY computation form and near-linear training throughput. At the 1.3B-parameter, 100B-token scale, it outperforms Mamba-2, Mamba-3, Gated DeltaNet, KDA, and Transformer baselines on language modeling, commonsense reasoning, and especially multi-key needle-in-a-haystack retrieval (89.8% at 2K context recurrent vs. KDA's 63.2%). Ablations show the erase gate contributes more than the write gate, and their gains are additive. Limitations include the lack of large-scale (70B+), ultra-long-context, and inference-latency evaluations.

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

| Paper | Details | |---|---| | Title | Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | | Authors | Ali Hatamizadeh, Yejin Choi, Jan Kautz | | Affiliation | NVIDIA Research | | arXiv | 2605.22791 | | Date | May 21, 2026 | | Code | github.com/NVlabs/GatedDeltaNet-2 |

Key points

Core idea: In delta-rule linear attention, memory *erasure* (along key directions) and memory *writing* (along value directions) are fundamentally different operations, yet prior models (Gated DeltaNet, KDA) tie them together with a single scalar gate β_t. Gated DeltaNet-2 splits them into two independent per-channel gates: an erase gate b_t ∈ [0,1]^{d_k} and a write gate w_t ∈ [0,1]^{d_v}.
Update rule:
Efficiency preserved: The per-channel gates are absorbed into the rank-one erasure factor, keeping the same chunkwise WY form as KDA (Triton kernels). On H100s, hybrid training throughput drops only slightly (≈38K → ≈36K tokens/s at seq length 2048), versus ≈25K for a standard Transformer that falls to ≈12K at 8K.

Results (1.3B params, 100B FineWeb-Edu tokens)

Language modeling / commonsense reasoning: Best average accuracy in both recurrent (53.11% vs. Mamba-3 MIMO 52.39%, KDA 52.28%) and hybrid settings (53.97% vs. KDA 52.68%). WikiText perplexity: 15.90 recurrent vs. KDA 16.81.
Long-context retrieval (RULER MK-NIAH): The most dramatic gains. Recurrent: 89.8% at 2K (KDA 63.2%, Mamba-3 MIMO 72.4%) and 37.8% at 4K (KDA 28.0%). Hybrid: 48.0% at 4K vs. KDA 40.4%, showing complementarity with sliding-window attention.
Real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP): Best averages in both settings (29.88% recurrent vs. KDA 28.67%; 42.28% hybrid vs. Mamba-3 SISO 41.01%).

Ablations

Reverting the erase gate to a scalar causes the largest drops; reverting the write gate to a scalar causes smaller but measurable drops.
The erase gate matters more than the write gate — deciding *what to delete* from a fixed-size state is more critical than how strongly to write. Their contributions are additive, not redundant.
Extending the erase gate range from [0,1] to [0,2] (over-erasure) yields no significant gain at 1.3B scale.

The bigger picture

The linear attention family shows a clear trajectory of increasingly fine-grained memory control: raw linear attention (no management) → Mamba-2's scalar decay (passive forgetting) → DeltaNet's scalar edit → KDA's per-channel decay → Gated DeltaNet-2's fully decoupled per-channel erase and write. Future directions may include communication between erase and write operations, or per-head erase-write strategies.

Caveats

All comparisons are at the 1.3B / 100B-token "fair small-scale" recipe; scaling to 70B+ is unvalidated.
The pure recurrent model was not tested on production-scale contexts (128K–1M tokens), though throughput results suggest linear scaling should hold.
No inference (decode) latency numbers are reported — only training throughput — though decode latency should be comparable to KDA's given the fixed-size state.

References

1. Hatamizadeh, A., Choi, Y., & Kautz, J. (2026). Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. *arXiv:2605.22791*. 2. Yang, S., et al. (2024). Gated DeltaNet. *arXiv:2412.06464*. 3. Kimi Team. (2025). KDA: Kimi Delta Attention. *arXiv:2510.26692*. 4. Dao, T., & Gu, A. (2024). Mamba-2: State Space Duality. *arXiv:2405.21060*. 5. Li, Y., et al. (2026). Mamba-3. *arXiv:2603.15569*.