LeapQuant: Training-Free 8-bit Quantization for Linear Attention Recurrent States

Forum topic · 小凯 · 2026-10-01

Summary

LeapQuant is a training-free quantization method for hybrid LLM architectures that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). These models compress context into a fixed-size recurrent state, but repeatedly reading and updating that state remains a major inference bottleneck, and naive quantization degrades quality due to rounding error accumulation and outlier rows/columns in the state. LeapQuant achieves near-lossless performance under 8-bit recurrent-state quantization through three techniques: per-window quantization that skips a token window and quantizes the state once at its end while computing outputs from a fixed low-bit state plus a high-precision buffer; retention of the largest outlier values as a small set of high-precision Compensator Tokens that share the update path with real tokens; and smoothing of the remaining residual before quantization. Experiments on Qwen, Kimi, and GLM model families show kernel-level speedups of 2.05-3.70x on average with accuracy comparable to FP32, and up to 1.47x end-to-end inference speedup on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs. Paper: arXiv 2609.38166.

Paper Overview

Field: Machine Learning

Authors: Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica

arXiv: 2609.38166

Abstract (translated)

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality due to the accumulation of rounding errors and the presence of outlier rows and columns in the state.

LeapQuant is a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization:

1. Per-window quantization: To mitigate error accumulation, it skips a window of tokens and quantizes the state only once at the end of the window. Within the window, outputs are computed from a fixed low-bit state plus a high-precision buffer update. 2. Compensator Tokens: To reduce per-quantization error, LeapQuant keeps the largest outlier values in the state as a small number of high-precision compensator tokens that share the update path with real tokens. 3. Residual smoothing: The remaining residual is smoothed to further reduce error before quantization.

Results

Evaluated on the Qwen, Kimi, and GLM model families
Accuracy comparable to the FP32 baseline
Kernel-level speedup of 2.05-3.70x on average
End-to-end inference speedup of up to 1.47x on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs