KDA: Kimi Delta Attention — A Linear Attention That Beats Standard Attention Across All Sequence Lengths (2025)

Forum topic · 小凯 · 2026-05-10

Summary

KDA (Kimi Delta Attention), introduced by the Kimi Team in 2025 (arXiv:2510.26692), is a hybrid linear attention architecture designed to overcome the O(n²) cost of standard attention while avoiding the quality loss that has historically plagued linear variants on short or complex sequences. Its core components include an enhanced Gated DeltaNet with fine-grained gating, a special Diagonal-Plus-Low-Rank transition matrix variant, a chunkwise algorithm that mixes local standard attention with global linear attention, and a layer-wise hybrid that combines KDA in shallow layers with MLA in deep layers. The 3B-active / 48B-total MoE model reports a 75% KV cache reduction and up to 6× higher 1M-context decode throughput. Under matched training recipes, KDA outperforms full MLA by a sizeable margin, making it the first linear attention shown to comprehensively surpass standard attention across all settings, including short context, long context, and RL scaling.

Overview

KDA (Kimi Delta Attention) is a hybrid linear attention architecture proposed by the Kimi Team in 2025 (arXiv: 2510.26692). It targets the central question of whether a linear attention mechanism can outperform standard quadratic attention across all scenarios — short context, long context, and reinforcement-learning scaling — rather than only excelling on long sequences while degrading elsewhere.

Core Problems Addressed

Quadratic cost of standard attention becomes prohibitive at long sequence lengths.
Prior linear attention variants (Performer, Linear Transformer) underperform standard attention on short sequences and complex tasks.
Open question: can linear attention become a general replacement for standard attention rather than a scenario-specific compromise?

Key Innovations

1. Gated DeltaNet with fine-grained gating — extends Yang's Gated DeltaNet with more refined gating so that the finite-state RNN memory is utilized more effectively. 2. Specialized DPLR transition matrix — uses a tailored Diagonal-Plus-Low-Rank transition matrix variant that substantially reduces compute while preserving consistency with the classical delta rule. 3. Chunkwise algorithm — splits the sequence into chunks; standard attention is applied within each chunk, and linear attention is applied between chunks. This balances local precision with global efficiency. 4. Inter-layer hybrid of KDA and MLA — shallow layers use KDA for efficiency, while deeper layers use MLA for precision, combining both at the model level.

Reported Numbers

Model size: 3B active parameters / 48B total parameters (MoE)
KV cache reduction: 75%
1M-context decode throughput: up to 6×
Under an identical training recipe, KDA outperforms full MLA by a sizeable margin
Open-sourced KDA kernel and vLLM implementation

Impact Assessment

KDA is described as the first linear attention that comprehensively surpasses standard attention under fair comparison. Earlier linear attention work only demonstrated advantages at long sequence lengths and lost to standard attention at short lengths. KDA breaks this scenario restriction, positioning linear attention as a general-purpose replacement rather than a niche optimization.

Analogy

Just as early electric sports cars were dismissed as lacking the sound and soul of combustion engines, linear attention was widely viewed as a speed-for-quality trade-off. KDA suggests that — given sufficiently clever gating and matrix decompositions — linear attention can win on every track, not just the long-sequence one. The caveat is that limits historically attributed to physical constraints are often actually artifacts of current technology.

Reference

arXiv: 2510.26692