Key points
Moonshot AI's Kimi team introduced Kimi Linear, a hybrid architecture the author frames as an "error-correcting dynamic memory" for LLMs, claiming it surpasses full attention across short-context, long-context, and RL settings while delivering major efficiency gains.
1. Kimi Delta Attention (KDA): error-correcting memory
- Standard Transformer full attention re-reads all context for every query, hitting a compute/memory wall at million-token scales. Linear attention keeps a fixed-size state but suffers from blurry, distorted accumulation.
- DeltaNet reframed linear attention as online gradient descent: the state update becomes
S_new = (I - βktkᵀ) * S_old + βktvᵀ— the model *corrects* the memory rather than only adding to it. - Gated DeltaNet (GDN) added a scalar forget gate that discounts the whole state uniformly. KDA's key upgrade is a channel-wise diagonal gate
Diag(αt): channels carrying long-range dependencies get αt near 1 (near-lossless retention), while channels holding transient details decay quickly, freeing memory bandwidth. - This fine-grained gating yields superior convergence and peak accuracy on synthetic retrieval tasks (Palindrome, MQAR) versus GDN and Mamba2.
- Despite higher expressiveness, KDA stays hardware-efficient via a diagonal-plus-low-rank (DPLR) representation and custom parallel kernels — roughly halving computation relative to similar fine-grained approaches like GLA, with ~2x kernel speedup at 64K sequence length.
- Ablations found a golden ratio: 3 KDA layers interleaved with 1 full-attention (MLA) layer. KDA layers act as domain experts handling local, efficient sequence processing; the periodic MLA layer performs global information synchronization. The 3:1 mix achieved lower training/validation loss than pure MLA (0:1) or 1:1 mixing.
- The MLA layers use no positional encoding (NoPE). KDA's data-dependent gating inherently provides positional/temporal awareness (recent information dominates; distant information is selectively kept or forgotten).
- Benefits: clear division of labor (KDA handles time/position, MLA handles global semantics) and smoother long-context extension without RoPE rescaling. NoPE variants beat RoPE variants on RULER and RepoQA.
- General knowledge/reasoning: best scores on MMLU and BBH; MMLU-Pro 51.0 vs MLA 47.2 and GDN-H 47.9.
- Math/code: on par or leading on GSM8K, MATH.
- Long context: RULER (128k) 84.3 vs MLA 81.3 / GDN-H 80.5; RepoQA 68.5 vs 63.0 for both baselines; best overall average of 54.5.
- RL fine-tuning: faster learning curves and higher ceiling than MLA on multi-step reasoning.
- Efficiency: up to 75% KV cache reduction (3/4 of layers are linear), higher throughput via larger batch sizes, and near-flat decoding latency — at 1M context, TPOT is 1.84ms vs 11.48ms for MLA (6.3x speedup).
2. Hybrid architecture: 3:1 ratio and NoPE
3. Empirical results
After equal-scale pretraining on 1.4T tokens, compared against full-attention MLA and GDN-H:
Conclusion
The author argues Kimi Linear shows linear attention is no longer a "cheap substitute" for full attention; that hybrid 3:1 mixing with a NoPE/full-attention division of labor is a validated design paradigm; and that it makes million-token applications practically usable. Moonshot AI has open-sourced the KDA kernel and a vLLM implementation to support community adoption.