Dynamic Linear Attention (DLA): Information-Aware State Merging Breaks Fixed-Chunking Limits in Linear Attention

Forum topic · 小凯 · 2026-06-11

Summary

Researchers from Ohio State University, University of Michigan, and ByteDance Seed propose Dynamic Linear Attention (DLA), which replaces the fixed multi-scale chunking of Log-Linear Attention with information-aware dynamic state merging. DLA computes a lightweight state information score (Frobenius-norm change rate) per token: semantically continuous tokens merge into the current state, while sharp semantic drifts trigger a new high-resolution state. A fixed-capacity cache (K=30) bounds memory via greedy merging of the lowest-information-density adjacent state pair. The paper proves fixed chunking's approximation error is dominated by within-block heterogeneity, which adaptive boundaries minimize. On Mamba-2-780M and Gated DeltaNet-1.3B backbones trained on 50B tokens, DLA outperforms Log-Linear Attention across 16 datasets: up to +52% on common-sense reasoning (LAMBADA), +49% on contextual retrieval, and +350% on multi-needle retrieval (MK-1). The Mamba-2 variant matches or beats a parameter-matched full-attention Transformer on 7 of 8 reasoning tasks, suggesting linear attention's ceiling lies in state organization rather than architecture. Efficiency is also improved, with higher throughput and controlled memory.

TL;DR: Dynamic Linear Attention (DLA), from Ohio State University, University of Michigan, and ByteDance Seed, replaces fixed multi-scale chunking in multi-state linear attention with information-aware dynamic state merging. Regions of sharp semantic change keep high resolution while stable regions are coarsely compressed. On Mamba-2-780M and Gated DeltaNet-1.3B backbones, DLA beats Log-Linear Attention across 16 datasets, and the Mamba-2 variant matches a parameter-matched full-attention Transformer — with up to 350% improvement on multi-needle retrieval.

1. The Problem: Fixed Chunking Kills Key Information

Linear attention reduces Transformer attention to linear complexity by compressing history into a single state matrix, limiting expressiveness. Log-Linear Attention (current SOTA multi-state method) uses fixed chunking via Fenwick tree decomposition: near history at high resolution, distant history coarsely.

The flaw: fixed chunking assumes uniform information density. Semantic shifts — a pivot word, a subject switch, a key entity — can occur anywhere. Fixed chunks force critical tokens into the same block as redundant neighbors, and since merging is irreversible, their independent contributions are permanently lost, with error accumulating over long sequences.

The paper proves this formally: the approximation bias of fixed chunking on non-stationary sequences is dominated by within-block heterogeneity. If a semantic change point falls inside a fixed block boundary, the bias is strictly larger than with adaptive boundaries.

2. DLA's Two Core Mechanisms

Mechanism 1: Information-Aware Dynamic State Merging

For each new token, DLA computes a lightweight State Information Score:

I_t = \frac{\|S_t - S_{t-1}\|_F}{\|S_{t-1}\|_F + \epsilon}

This measures the Frobenius-norm change rate of the state representation. Low score = semantic continuity → merge into current state; high score = significant drift → create a new independent state. This yields adaptive resolution allocation: chit-chat transitions are heavily compressed, sudden technical terms or subject switches get new high-resolution states, and chapter boundaries split naturally.

Training uses a soft (differentiable) gate; inference switches to hard splits for discrete states aligned with semantic boundaries.

Mechanism 2: Capacity-Constrained Memory Management

Unconstrained merging would grow memory unboundedly. DLA maintains a fixed-size K cache of chronologically ordered states (K=30, matching Log-Linear Attention's max state count). When full, it triggers selective merging of the adjacent pair with lowest information density:

(i^*, i^*+1) = \arg\min_{i} \frac{\bar{I}_i + \bar{I}_{i+1}}{n_i + n_{i+1}}

where $\bar{I}$ is the sum of token information scores in a state and $$n$$ the token count. This merges only adjacent states (preserving temporal semantics), prefers information-sparse regions (minimizing information loss), and guarantees bounded compute and memory.

3. Theoretical Guarantee: Why Dynamic Beats Fixed

For exact linear attention output $$y(q)$$ vs. chunked summary output $\tilde{y}_\pi(q)$ :

\|y(q) - \tilde{y}_\pi(q)\| \leq \|q\|_2 \cdot \sum_{i=1}^m \sqrt{|\mathcal{C}_i|} \cdot \sqrt{\sum_{t \in \mathcal{C}_i} \|u_t - \bar{u}_i\|_2^2}

The bound is dominated by within-block heterogeneity. Fixed, content-agnostic chunking mixes tokens from different semantic regions, giving a strictly positive heterogeneity term; adaptive alignment with semantic change points makes it zero. DLA's information-aware merging is a greedy online policy that approximately minimizes this dominant term.

4. Experiments: Leading Across 16 Datasets

Setup: Mamba-2-780M and Gated DeltaNet-1.3B backbones; 50B tokens (Long-Data-Collections), 16K sequence length; baselines: vanilla linear attention, Log-Linear Attention, and a parameter-matched 778M full-attention Transformer; 4x NVIDIA A100.

Common-Sense Reasoning (8 datasets)

| Model | Backbone | LAMBADA | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | CSQA | |---|---|---|---|---|---|---|---|---|---| | Log-Linear | Mamba-2 | 37.57 | 64.58 | 40.37 | 54.30 | 53.49 | 32.51 | 40.80 | 26.52 | | DLA | Mamba-2 | 57.16 | 69.10 | 43.30 | 55.96 | 54.54 | 34.81 | 42.80 | 28.02 | | Transformer | — | 42.08 | 67.03 | 42.39 | 55.01 | 53.66 | 33.02 | 42.20 | 27.40 | | Log-Linear | DeltaNet | 42.30 | 68.77 | 43.24 | 55.96 | 55.18 | 36.26 | 44.80 | 28.31 | | DLA | DeltaNet | 45.19 | 69.37 | 44.39 | 56.59 | 55.52 | 36.69 | 45.40 | 28.80 |

DLA improves over Log-Linear by up to +52% on Mamba-2 (LAMBADA) and +22% on Gated DeltaNet. Notably, Mamba-2 + DLA beats the parameter-matched full-attention Transformer on 7 of 8 tasks — the ceiling is state organization, not architecture.

Contextual Retrieval (6 datasets)

On SWDE, SQuAD, FDA, TriviaQA, Drop, and NQ, DLA consistently leads, with up to +49% relative improvement.

Long-Context Modeling (RULER + LongBench)

| Task | Type | Vanilla | Log-Linear | DLA | Relative Gain | |---|---|---|---|---|---| | S-NIAH-1 | Single-needle | 62.50 | 74.50 | 84.00 | +13% | | S-NIAH-2 | Single-needle | 65.00 | 72.00 | 78.50 | +9% | | S-NIAH-3 | Single-needle | 46.00 | 52.00 | 63.50 | +22% | | MK-1 | Multi-needle | 0.00 | 5.50 | 24.75 | +350% | | MQ | Multi-query | 3.00 | 15.00 | 25.00 | +67% | | MV | Multi-value | 15.75 | 24.75 | 31.75 | +28% |

Multi-needle tasks require simultaneously locating and associating multiple scattered pieces of information. DLA jumps from 5.50% to 24.75% on MK-1. On LongBench, DLA leads across narrative QA, multi-field QA, summarization, and few-shot learning, showing the gains generalize beyond retrieval.

5. Efficiency

Higher throughput: dynamic merging avoids unnecessary computation
Lower memory: the fixed-capacity cache prevents linear growth with sequence length
Predictable cost: the capacity cap K bounds worst-case complexity for deployment planning

6. Why It Matters

Prior linear attention work (Mamba, DeltaNet, GLA) focused on kernel approximations and recurrent state updates, not on how states are organized. DLA's contributions:

1. Identifies the real bottleneck: not state size, but state boundary intelligence 2. Scalable solution: information-aware merging + capacity control, both lightweight with no added inference complexity 3. Theory + empirics: a theorem on fixed chunking's suboptimality plus validation on 16 datasets 4. Generality: works on two very different backbones (Mamba-2, Gated DeltaNet)

For long-context applications (document analysis, codebase understanding, multi-turn dialogue, video modeling), DLA offers a practical way to improve long-range modeling without scaling model size.

7. Limitations and Future Work

1. Capacity K is a hyperparameter (fixed at 30); dynamic or task-adaptive K is future work 2. Only output supervision explored; per-step attention weight analysis is lacking 3. Scale untested beyond 780M–1.3B; validity at 7B/13B remains open 4. Shares max state count with Log-Linear for fair comparison; larger capacity could widen DLA's advantage

References

Wang, X., Shen, H., Zheng, B., Liu, X., Cho, M., Wan, Z., Zhao, Z., Mao, Z., Yan, S., & Zhang, M. (2026). *Dynamic Linear Attention*. arXiv:2606.10650. https://arxiv.org/abs/2606.10650
Dao, T., & Gu, A. (2024). *Mamba-2*. ICML 2024.
Yang, S., et al. (2024). *DeltaNet*. NeurIPS 2024.
Li, X., et al. (2025). *Log-Linear Attention*. arXiv:2506.04761.