Dynamic Linear Attention: Information-Aware State Merging Beats Fixed Chunking in Linear Attention

Forum topic · 小凯 · 2026-06-11

Summary

Researchers from Ohio State University, University of Michigan, and ByteDance Seed propose Dynamic Linear Attention (DLA), which replaces the fixed multi-scale chunking of Log-Linear Attention with information-aware dynamic state merging. DLA computes a State Information Score per token: low scores are merged into the current state, while high scores create new states, allocating high resolution to semantically volatile regions and coarse compression to stable ones. A capacity-limited, time-ordered state cache (K=30) merges the lowest-density adjacent state pair when full, keeping memory and compute bounded. The paper proves that approximation error in chunked linear attention is dominated by within-block heterogeneity, which adaptive boundaries minimize. On Mamba-2-780M and Gated DeltaNet-1.3B backbones trained on 50B tokens, DLA outperforms Log-Linear Attention across 16 benchmarks, with up to 52% relative gain on LAMBADA and a 350% improvement on multi-needle retrieval (RULER MK-1). The Mamba-2 variant matches or exceeds a parameter-matched full-attention Transformer, suggesting that state organization—not architecture size—is the key bottleneck for linear attention in long-context modeling.

TL;DR: Dynamic Linear Attention (DLA), from Ohio State University, University of Michigan, and ByteDance Seed, replaces fixed multi-scale chunking in multi-state linear attention with information-aware dynamic state merging. Semantically volatile regions keep high-resolution states; stable regions are coarsely compressed. On Mamba-2-780M and Gated DeltaNet-1.3B, DLA beats Log-Linear Attention across 16 datasets, and the Mamba-2 variant matches a parameter-matched full-attention Transformer. Gains on the hardest multi-needle retrieval task reach 350%.

The Problem: Fixed Chunking Destroys Key Information

Linear attention reduces Transformer quadratic complexity to linear, but compresses the entire history into a single state matrix, limiting expressivity. Log-Linear Attention (current SOTA multi-state method) uses fixed chunking via a Fenwick tree decomposition: log-number of multi-scale states, high resolution near, coarse far.

The flaw: fixed chunking assumes uniform information density. Semantic shifts—topic changes, entity introductions—can occur anywhere, and fixed chunks force critical tokens into the same block as redundant neighbors, irreversibly merging them. The paper proves that the approximation bias of fixed chunking on non-stationary sequences is dominated by within-block heterogeneity: if a semantic change point falls inside a fixed block boundary, the bias is strictly larger than with adaptive boundaries.

DLA's Two Core Mechanisms

1. Information-Aware Dynamic State Merging

For each new token, DLA computes a lightweight State Information Score:

I_t = \frac{\|S_t - S_{t-1}\|_F}{\|S_{t-1}\|_F + \epsilon}

Low score = semantic continuity → merge into current state. High score = significant drift → create a new state. This yields adaptive resolution allocation: chit-chat transitions get heavily compressed; technical terms or subject switches get fresh high-resolution states; document section boundaries split naturally. Training uses soft (differentiable) gating; inference switches to hard splitting.

2. Capacity-Limited Memory Management

DLA maintains a fixed-size, time-ordered state cache (K=30 in experiments, matching Log-Linear Attention's max state count). When full, it merges the adjacent state pair with the lowest information density:

(i^*, i^*+1) = \arg\min_{i} \frac{\bar{I}_i + \bar{I}_{i+1}}{n_i + n_{i+1}}

Only adjacent states merge (preserving temporal semantics), and information-sparse regions merge first (minimizing loss), keeping compute and memory strictly bounded.

Theoretical Guarantee

For exact output $$y(q)$$ and chunked output $\tilde{y}_\pi(q)$ :

\|y(q) - \tilde{y}_\pi(q)\| \leq \|q\|_2 \cdot \sum_{i=1}^m \sqrt{|\mathcal{C}_i|} \cdot \sqrt{\sum_{t \in \mathcal{C}_i} \|u_t - \bar{u}_i\|_2^2}

The bound is dominated by within-block heterogeneity. Fixed chunking mixes tokens from different semantic regions (strictly positive heterogeneity); adaptive boundaries aligned to semantic change points drive it to zero. DLA's merging acts as a greedy online policy approximately minimizing this term.

Experiments: Leading Across 16 Datasets

Backbones: Mamba-2-780M, Gated DeltaNet-1.3B; trained on 50B tokens (Long-Data-Collections), 16K sequence length; 4× NVIDIA A100.
Baselines: Vanilla linear attention, Log-Linear Attention, parameter-matched full-attention Transformer-778M (24 layers).
Tasks: 8 commonsense reasoning, 6 context retrieval, 2 long-context modeling benchmarks.

Commonsense Reasoning (8 datasets)

| Model | Backbone | LAMBADA | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | CSQA | |-------|----------|---------|------|-----------|------------|-------|-------|------|------| | Vanilla | Mamba-2 | 34.48 | 64.58 | 37.36 | 53.20 | 53.03 | 30.29 | 39.80 | 24.57 | | Log-Linear | Mamba-2 | 37.57 | 64.58 | 40.37 | 54.30 | 53.49 | 32.51 | 40.80 | 26.52 | | DLA | Mamba-2 | 57.16 | 69.10 | 43.30 | 55.96 | 54.54 | 34.81 | 42.80 | 28.02 | | Transformer | — | 42.08 | 67.03 | 42.39 | 55.01 | 53.66 | 33.02 | 42.20 | 27.40 | | Vanilla | DeltaNet | 42.30 | 68.77 | 43.23 | 55.88 | 55.01 | 35.84 | 44.40 | 28.02 | | Log-Linear | DeltaNet | 42.30 | 68.77 | 43.24 | 55.96 | 55.18 | 36.26 | 44.80 | 28.31 | | DLA | DeltaNet | 45.19 | 69.37 | 44.39 | 56.59 | 55.52 | 36.69 | 45.40 | 28.80 |

DLA gains up to 52% over Log-Linear on Mamba-2 (LAMBADA) and up to 22% on DeltaNet. Crucially, Mamba-2 + DLA beats the parameter-matched full-attention Transformer on 7 of 8 tasks—the ceiling of linear attention is the state organization strategy, not the architecture.

Context Retrieval (6 datasets)

On SWDE, SQuAD, FDA, TriviaQA, Drop, and NQ, DLA consistently leads, with up to 49% relative improvement.

Long-Context Modeling (RULER + LongBench)

| Task | Type | Vanilla | Log-Linear | DLA | Relative Gain | |------|------|---------|------------|-----|---------------| | S-NIAH-1 | single-needle | 62.50 | 74.50 | 84.00 | +13% | | S-NIAH-2 | single-needle | 65.00 | 72.00 | 78.50 | +9% | | S-NIAH-3 | single-needle | 46.00 | 52.00 | 63.50 | +22% | | MK-1 | multi-needle | 0.00 | 5.50 | 24.75 | +350% | | MQ | multi-needle | 3.00 | 15.00 | 25.00 | +67% | | MV | multi-needle | 15.75 | 24.75 | 31.75 | +28% |

On multi-needle retrieval—requiring locating and relating multiple dispersed facts—DLA jumps from 5.50% to 24.75% on MK-1. On LongBench, DLA leads across narrative QA, multi-field QA, summarization, and few-shot learning, showing the benefit generalizes beyond retrieval.

Efficiency

Higher throughput: dynamic merging reduces unnecessary computation.
Lower memory: fixed-capacity cache prevents unbounded growth.
Predictable cost: the capacity cap K guarantees worst-case complexity for deployment planning.

Why It Matters

Prior linear attention work (Mamba, DeltaNet, GLA) focused on kernel approximation and recurrent state updates, neglecting how states are organized. DLA's contributions:

1. Identifies the real bottleneck: not state size, but state boundary intelligence. 2. A scalable solution: information-aware merging + capacity control, both lightweight with no inference overhead. 3. Dual theoretical and empirical validation. 4. Generalizes across two very different backbones (Mamba-2, Gated DeltaNet).

Limitations

1. K is a hyperparameter (fixed at 30); adaptive capacity remains unexplored. 2. Only output-quality supervision analyzed; no per-step attention analysis. 3. Not validated beyond 780M–1.3B scale. 4. Sharing the max state count with Log-Linear (for fair comparison) may understate DLA's potential.

References

Wang, X., Shen, H., Zheng, B., Liu, X., Cho, M., Wan, Z., Zhao, Z., Mao, Z., Yan, S., & Zhang, M. (2026). *Dynamic Linear Attention*. arXiv:2606.10650. https://arxiv.org/abs/2606.10650
Dao, T., & Gu, A. (2024). *Mamba-2*. ICML 2024.
Yang, S., et al. (2024). *DeltaNet*. NeurIPS 2024.
Li, X., et al. (2025). *Log-Linear Attention*. arXiv:2506.04761.