Paper Overview
Field: NLP Authors: Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti Published: 2026-05-19 arXiv: 2505.14310
Summary
Current hierarchical attention methods, such as NSA and InfLLMv2, select the top-k relevant key-value (KV) blocks based on coarse attention scores and subsequently apply fine-grained softmax attention on the selected tokens. However, the top-k operation assumes the number of relevant tokens for any query is fixed, and it precludes the gradient flow between the sparse and dense stages.
In this work, the authors propose DashAttention (Differentiable and Adaptive Sparse Hierarchical Attention), which leverages the adaptively sparse \(\alpha\)-entmax transformation to select a variable number of blocks according to the current query in the first stage. This in turn provides a prior for the second-stage softmax attention, keeping the entire hierarchy fully differentiable.
Key Results
- Unlike other hierarchical attention methods, DashAttention is proven to be non-scattered, leading to better long-context modeling capability.
- Experiments on large language models show DashAttention achieves accuracy comparable to full attention at 75% sparsity.
- It demonstrates a better Pareto frontier than NSA and InfLLMv2 in the high-sparsity regime.
- An efficient Triton-based, GPU-aware implementation is provided, achieving faster inference than FlashAttention-3.
*Auto-collected on 2026-05-20*