Gated DeltaNet-2: NVIDIA Decouples Erase and Write in Linear Attention

Paper: *Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention* Authors: Ali Hatamizadeh, Yejin Choi, Jan Kautz (NVIDIA Research) arXiv: 2605.22791 | Code: github.com/NVlabs/GatedDeltaNet-2 Date: May 21, 2026 | Category: cs.AI

Key points

• Core idea: In linear attention with delta-rule memory updates, erasing old memory (along key directions) and writing new memory (along value directions) are fundamentally different operations acting on different axes. Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) tie them together with a single scalar gate β_t. Gated DeltaNet-2 splits them into two independent per-channel gates — like letting one hand hold the eraser and the other hold the chalk, each with its own fine-grained control.
• The update rule: With an erase gate b_t ∈ [0,1]^{d_k} and a write gate w_t ∈ [0,1]^{d_v}:
\[\mathbf{S}_t = (I - k_t \cdot (\mathbf{b}_t \odot k_t)^\top) \cdot \mathbf{D}_t \cdot \mathbf{S}_{t-1} + k_t \cdot (\mathbf{w}_t \odot v_t)^\top\]

Per-channel decay D_t is inherited from KDA; the erase term now applies directional per-key-channel weighting, and the write term weights values per-channel. Where KDA's scalar β_t has 1 degree of freedom, GDN-2 has d_k + d_v (e.g., 256 for d_k = d_v = 128).

• Background: Linear attention compresses history into a fixed-size state matrix updated per token, avoiding O(n²) attention cost. The field evolved from scalar decay (Mamba-2), to delta-rule editing (DeltaNet), to combining both (Gated DeltaNet), to per-channel decay (KDA). GDN-2 completes this trajectory by fully decoupling erase from write.
• Efficiency: Training throughput on H100 drops only slightly (≈38K → ≈36K tokens/s at seq length 2048), far above a standard Transformer (≈25K, falling to ≈12K at 8K). Per-channel decay is absorbed into the rank-one erase update, preserving KDA's chunkwise WY form with optimized Triton kernels.
• Results (1.3B params, 100B FineWeb-Edu tokens):
• Language modeling / commonsense reasoning (10-task average): 53.11% recurrent, 53.97% hybrid — beating Mamba-3 MIMO (52.39%) and KDA (52.28% / 52.68%). WikiText perplexity 15.90 (recurrent) vs. KDA's 16.81.
• Long-context retrieval (RULER): multi-key needle-in-a-haystack (MK-NIAH) at 2K context: 89.8% vs. Mamba-3 MIMO 72.4% and KDA 63.2%; at 4K: 37.8% recurrent, 48.0% hybrid vs. KDA 28.0%/40.4%.
• Real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP): leads across the board (29.88% recurrent, 42.28% hybrid averages).
• Ablations: Reverting either gate to a scalar hurts performance. The erase gate contributes more than the write gate — what gets deleted matters more than how strongly new data is written in a fixed-size state. The two contributions are additive, not redundant. Extending the erase gate range beyond [0,1] to [0,2] yields no gain at 1.3B scale.

Caveats

• Comparisons use a small-scale (1.3B/100B) "fair recipe"; scaling behavior at 70B+ is unverified.
• The pure recurrent version was not tested on true production-scale contexts (128K–1M tokens), though throughput experiments confirm linear scaling in principle.
• No inference (decode) latency numbers are reported — only training throughput — though decode latency should be in the same ballpark as KDA.

The paper's contribution is a conceptual clarification as much as a SOTA result: erase and write are different operations and should be controlled separately. Natural next steps include communication between the two gates and per-head erase-write policies.