STEPQuant: Spatial-Temporal Post-Training Quantization for Delta-Rule Recurrent States

Forum topic · 小凯 · 2026-10-01

Summary

STEPQuant (arXiv:2609.38169) is a post-training quantization framework for linear-attention recurrent states, addressing a key memory bottleneck in concurrent LLM serving. While linear attention replaces growing KV caches with fixed-size recurrent states, quantizing these states to low precision typically causes severe accuracy loss because quantization errors propagate through successive state updates. The authors analyze error impact along two dimensions: temporally, errors in long-lived memory persist across many decoding steps; spatially, different key rows affect model outputs differently, with state magnitudes varying along both rows and columns. STEPQuant allocates precision based on error magnitude and memory lifespan, and jointly fits scaling factors for key rows and value columns informed by state distributions and output error sensitivity. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across long- and short-generation benchmarks show that 6-bit STEPQuant closely matches FP32 state accuracy, and its 4-bit configuration outperforms uniform INT8. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent state compression and reduces total serving memory by up to 68.7%. Code is open-sourced.

Paper Overview

Field: NLP Authors: Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei Release date: 2026-09-29 arXiv: 2609.38169

English Translation of the Post Summary

Linear attention replaces growing KV caches with fixed-size recurrent states, but under concurrent serving, these persistent states can become a significant memory bottleneck. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, because quantization errors propagate through successive state updates. The authors find that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns.

Based on these observations, the authors propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifespan, and jointly fits scaling factors for key rows and value columns based on state distributions and the impact of key rows on output errors.

Results

Evaluated on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across long- and short-generation benchmarks
Under a nominal 6-bit budget, STEPQuant closely matches FP32 state accuracy
Its 4-bit configuration outperforms uniform INT8
Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent state compression and reduces total serving memory by up to 68.7%

The code has been open-sourced.

Original Abstract (excerpt)

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates...

---

*Automatically collected on 2026-10-01*