KV cache is a memory nightmare during LLM inference: with long sequences it can reach tens of GB, often several times the size of the model weights themselves. Quantizing each value from 16 bits down to 2 bits is the direct fix — but until now, no one had made 2-bit quantization preserve accuracy.
Simple rotations (such as the Hadamard transform) reduce outliers, but at 2 bits accuracy still collapses to near zero. Zhou, Zhuang, Li, Chen, Song, Athiwaratkun, and Wu found the reason: the rotation is not aligned with the covariance structure that attention actually consumes — the quantized KV distribution mismatches what attention needs to see.
OSCAR estimates, offline, the covariance structure that each attention head actually consumes at inference time, and uses this information to derive a fixed rotation matrix and clipping thresholds. The entire process requires no online computation — the covariance is computed from sampled calibration data before deployment, and the rotation matrix is fixed. They then wrote a custom INT2 attention CUDA kernel, fully compatible with paged KV cache and fused-kernel pipelines, which drops directly into SGLang and vLLM.
The results are convincing. On Qwen3-4B and Qwen3-8B, OSCAR's 2-bit accuracy is only 1.4–3.8 points below BF16, while naive INT2 with simple rotation approaches zero accuracy. Performance stays on par with BF16 on Qwen3-32B and GLM-4.7 (358B parameters) as well. Long-context robustness holds too (128K RULER-NIAH). On the systems side: roughly 8x reduction in KV cache memory, 7x throughput improvement at large batch sizes, and 3x faster single-sequence decoding.
Unclear points: how much calibration data is needed for the offline covariance estimation? How much does covariance structure vary across task types — if the calibration data used differs from the task distribution at inference time, does the rotation matrix still hold? MFU numbers for the INT2 kernel on the 358B model are not disclosed.
---
References
1. Zhou, Z., Zhuang, D., Li, J., et al. (2026). *OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization*. arXiv:2605.17757 [cs.LG].
2. Dao, T., et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. NeurIPS.
3. Ashkboos, S., et al. (2024). *QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks*. ICML.