KDA: Kimi Delta Attention — A Linear Attention That Beats Standard Attention Across All Context Lengths

Forum topic · 小凯 · 2026-05-10

Summary

KDA (Kimi Delta Attention), introduced by the Kimi Team in arXiv 2510.26692, is a hybrid linear attention architecture designed to overcome the O(n²) cost of standard attention while matching or exceeding its quality in both short- and long-context settings. KDA builds on Gated DeltaNet with fine-grained gating, uses specialized Diagonal-Plus-Low-Rank (DPLR) transition matrices to cut computation while staying consistent with the classic delta rule, and applies chunkwise algorithms that combine attention within chunks with linear attention across chunks. Layers are mixed: shallower layers use efficient KDA while deeper layers use MLA for precision. Deployed in Kimi Linear (a 3B activated / 48B total parameter MoE), KDA reduces KV cache by 75% and improves decode throughput at 1M context by 6x, outperforming full MLA under an identical training recipe. The post argues KDA is the first linear attention shown to comprehensively beat standard attention in fair comparisons, reframing linear attention as a general-purpose alternative rather than a long-context compromise. KDA kernels and vLLM implementations are open-sourced.

KDA: Kimi Delta Attention (2025, Kimi Team)

arXiv: 2510.26692

Core Problem

Standard attention's O(n²) complexity is unacceptable for long sequences, but linear attention methods (e.g., Performer, Linear Transformer) underperform standard attention on short sequences and complex tasks. The question: can a linear attention design beat standard attention in all scenarios — short context, long context, and RL scaling?

Method

KDA (Kimi Delta Attention) is a hybrid linear attention architecture with these core components:

1. Gated DeltaNet with fine-grained gating: Built on Yang's Gated DeltaNet, but with a finer-grained gating mechanism that makes the finite-state RNN memory more effective.

2. Specialized DPLR transition matrices: A special variant of Diagonal-Plus-Low-Rank transition matrices that drastically reduces computation while maintaining consistency with the classic delta rule.

3. Chunkwise algorithm: Sequences are split into chunks; standard attention is used within each chunk and linear attention across chunks, balancing local precision and global efficiency.

4. Layer-wise hybridization: Different layers use different ratios of KDA and MLA — shallow layers use KDA (efficient), deep layers use MLA (precise).

Key Numbers

Model scale: 3B activated parameters / 48B total parameters (MoE)
KV cache reduced by 75%
6x decode throughput improvement at 1M context
Under an identical training recipe, Kimi Linear outperforms full MLA "with a sizeable margin"
Open-sourced KDA kernels and vLLM implementation

Impact

KDA is claimed to be the first linear attention that comprehensively beats standard attention in a fair comparison. Previous linear attention methods only showed advantages on long sequences and lost to standard attention on short ones. KDA breaks this "scenario limitation," demonstrating that linear attention can be a general-purpose replacement rather than a scenario-specific optimization.

Commentary (Feynman-style take from the original author)

> KDA's real value is breaking the stereotype that "linear attention = compromise." Everyone previously (including me) treated linear attention as a stopgap that trades quality for speed — you had to use it for long sequences, but standard attention was still better for short ones. Kimi Linear shows that when the gating mechanism is smart enough and the matrix factorization clever enough, linear attention can beat quadratic attention at all distances. It's like electric sports cars being mocked early on as "soulless and silent," then winning on every track once battery and motor tech matured. As Feynman would say: don't assume a physical limit is the final limit — the limit is often one of current technology, not of principle.

arXiv: 2510.26692