Gated DeltaNet-2: Decoupling Erase and Write Gates in Linear Attention for Better Memory Editing

Forum topic · 小凯 · 2026-05-22

Summary

Gated DeltaNet-2, from NVIDIA researchers Ali Hatamizadeh, Yejin Choi, and Jan Kautz, addresses a key limitation of linear attention models: a single scalar gate controls both erasing old associations and writing new ones into the fixed-size recurrent state. The new Gated Delta Rule-2 splits this scalar into two independent channel-wise gates—an erase gate acting on the key side and a write gate acting on the value side—strictly generalizing Gated DeltaNet and KDA. Trained uniformly at 1.3B parameters on 100B FineWeb-Edu tokens, the model achieves the best results among pure recurrent and hybrid models on language modeling, common-sense reasoning, and especially long-context retrieval: 72.6% on multi-key needle-in-a-haystack at 1K versus 54.0% for KDA. Ablations show the erase gate's channel-wise structure contributes more than the write gate's. The method retains chunkwise parallel training via a WY representation, adding only negligible overhead on H100 GPUs.

The Memory Dilemma

Linear attention models compress entire sequence history into a fixed-size matrix state S, achieving linear-time sequence mixing and constant decoding memory—but the fixed state means new information must overwrite old information. How that overwrite is controlled is the central design question.

The evolution so far:

Basic linear attention: only adds. Each new token's key-value outer product accumulates into the state, with no forgetting mechanism.
Mamba-2: introduces data-dependent scalar decay α ∈ (0,1], multiplying old state before each write—global forgetting.
DeltaNet: applies the Delta Rule—read out the old value with the current key, subtract it, then write the new value. Precise overwrite.
Gated DeltaNet: combines scalar decay with the Delta Rule.
KDA (Kimi Delta Attention): upgrades scalar decay to channel-wise decay.

The Scalar Bottleneck

Even in KDA, the Delta Rule's editing gate β_t remains a scalar controlling two fundamentally different operations:

1. Erasing: how much old content to remove — a *key-side* decision about which coordinates of old associations to clear. 2. Writing: how much new content to commit — a *value-side* decision about which coordinates of the new value to store.

One knob for both dimensions is like one volume key controlling both stereo channels.

Gated Delta Rule-2

Gated DeltaNet-2 splits β_t into two independent channel-wise gates:

Erase gate b_t ∈ [0,1]^{d_k}: key-side, per-channel degree of removal of old content
Write gate w_t ∈ [0,1]^{d_v}: value-side, per-channel degree of committing new values

Update rule:

S_t = (I - k_t(b_t ⊙ k_t)^⊤) D_t S_{t-1} + k_t(w_t ⊙ v_t)^⊤

When b_t = β_t·1 and w_t = β_t·1, the model reduces to KDA; with scalar decay too, it reduces to Gated DeltaNet—so Gated DeltaNet-2 is a strict generalization.

From the fast-weight perspective, Gated Delta Rule-2 solves an online optimization problem:

S_t = argmin_S ‖S - S̄_t‖²_F - 2⟨S^⊤ k_t, z_t - S̄_t^⊤ e_t⟩

where z_t = w_t ⊙ v_t is the gated write target and e_t = b_t ⊙ k_t is the gated readout—selectively erasing only what needs removal and writing only what needs persistence.

Experiments (1.3B params, 100B FineWeb-Edu tokens)

Language modeling and reasoning

| Model | WikiText ppl↓ | LAMBADA ppl↓ | Commonsense Avg↑ | |------|:---:|:---:|:---:| | Mamba-2 | 16.79 | 12.38 | 51.82 | | Gated DeltaNet | 16.40 | 11.89 | 52.07 | | KDA | 16.81 | 11.68 | 52.28 | | Mamba-3 (MIMO) | 16.45 | 11.66 | 52.39 | | Gated DeltaNet-2 | 15.90 | 11.41 | 53.11 | | Hybrid (+SWA) Gated DeltaNet-2 | 15.62 | 10.43 | 53.97 |

Since recurrent state sizes match, gains come from the stronger update rule, not larger memory.

Long-context retrieval (RULER, MK-NIAH-1)

| Model | @1K | @2K | @4K | |------|:---:|:---:|:---:| | Mamba-2 | 29.0 | 21.2 | 21.4 | | Gated DeltaNet | 58.0 | 37.0 | 27.8 | | KDA | 54.0 | 44.2 | 28.0 | | Gated DeltaNet-2 | 72.6 | 51.4 | 37.8 |

Gated DeltaNet-2 leads by nearly 19 points over KDA at 1K context; with hybrid (sliding-window attention), 4K MK-NIAH reaches 48.0%. It also achieves the best averages on real-world retrieval tasks (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP): 29.88% pure recurrent, 42.28% hybrid.

Ablation: the erase gate matters more

Degating one side back to a scalar while keeping the other channel-wise:

| Variant | Commonsense Avg | MK-NIAH @4K | Retrieval Avg | |------|:---:|:---:|:---:| | Channel-wise write gate only | 52.45 | 30.6 | 28.92 | | Channel-wise erase gate only | 52.79 | 35.2 | 29.51 | | Full Gated DeltaNet-2 | 53.11 | 37.8 | 29.88 |

Both gates exploit channel freedom, but the erase gate's channel structure contributes more—knowing what to forget appears more important than knowing what to remember.

Engineering Insights

1. Decoupling is nearly free: splitting one scalar into two channel-wise gates adds just two projection matrices W_b and W_w, with significant gains. 2. Chunkwise parallel training still works: the paper derives a WY representation and gate-aware backward pass, absorbing channel-wise gating into asymmetric erasure factors—compatible with existing efficient delta-rule kernels, with only minor constant overhead vs. KDA (throughput 38.0 → 36.1 Kt/s on H100). 3. Hybrid is the practical route: pure recurrent models still lag at long contexts (37.8% at 8K MK-NIAH), but adding 2K sliding-window attention lifts 4K MK-NIAH to 48.0%. 4. Erasing beats writing: if you can only refine one gate channel-wise, choose the erase gate.

The broader lesson: when a single gate controls two independent mechanisms, split them—the cost is minimal and the gains, especially in memory-interference-heavy settings like multi-key retrieval, can be substantial.

Paper: Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Code: github.com/NVlabs/GatedDeltaNet-2

Authors: Ali Hatamizadeh, Yejin Choi, Jan Kautz (NVIDIA)