English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

OSCAR: 2-bit KV Cache Quantization via Spectral Covariance-Aware Rotation

Forum topic · 小凯 · 2026-05-19

Summary

OSCAR is a method for 2-bit quantization of the KV cache in LLM inference, addressing the memory bottleneck where long-sequence KV caches can reach tens of GB. Prior approaches using simple rotations (e.g., Hadamard transforms) reduce outliers but collapse to near-zero accuracy at 2 bits. The authors identify the root cause: rotation is not aligned with the covariance structure that attention actually consumes. OSCAR estimates each attention head's covariance structure offline from calibration data, derives a fixed rotation matrix and clipping thresholds from it, and requires no online computation. A custom INT2 attention CUDA kernel is fully compatible with paged KV cache and fused kernel pipelines, integrating directly into SGLang and vLLM. On Qwen3-4B/8B, OSCAR's 2-bit accuracy is only 1.4-3.8 points below BF16, while naive INT2 rotation approaches zero accuracy; results hold on Qwen3-32B and the 358B-parameter GLM-4.7, and remain robust on 128K-context RULER-NIAH. System gains: ~8x KV cache memory reduction, 7x throughput at large batch sizes, and 3x single-sequence decoding speedup. Open questions include calibration data requirements, robustness to task distribution shift, and undisclosed MFU on the 358B model.

KV cache is a memory nightmare during LLM inference: with long sequences it can reach tens of GB, often several times the size of the model weights themselves. Quantizing each value from 16 bits down to 2 bits is the direct fix — but until now, no one had made 2-bit quantization preserve accuracy.

Simple rotations (such as the Hadamard transform) reduce outliers, but at 2 bits accuracy still collapses to near zero. Zhou, Zhuang, Li, Chen, Song, Athiwaratkun, and Wu found the reason: the rotation is not aligned with the covariance structure that attention actually consumes — the quantized KV distribution mismatches what attention needs to see.

OSCAR estimates, offline, the covariance structure that each attention head actually consumes at inference time, and uses this information to derive a fixed rotation matrix and clipping thresholds. The entire process requires no online computation — the covariance is computed from sampled calibration data before deployment, and the rotation matrix is fixed. They then wrote a custom INT2 attention CUDA kernel, fully compatible with paged KV cache and fused-kernel pipelines, which drops directly into SGLang and vLLM.

The results are convincing. On Qwen3-4B and Qwen3-8B, OSCAR's 2-bit accuracy is only 1.4–3.8 points below BF16, while naive INT2 with simple rotation approaches zero accuracy. Performance stays on par with BF16 on Qwen3-32B and GLM-4.7 (358B parameters) as well. Long-context robustness holds too (128K RULER-NIAH). On the systems side: roughly 8x reduction in KV cache memory, 7x throughput improvement at large batch sizes, and 3x faster single-sequence decoding.

Unclear points: how much calibration data is needed for the offline covariance estimation? How much does covariance structure vary across task types — if the calibration data used differs from the task distribution at inference time, does the rotation matrix still hold? MFU numbers for the INT2 kernel on the 358B model are not disclosed.

---

References

1. Zhou, Z., Zhuang, D., Li, J., et al. (2026). *OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization*. arXiv:2605.17757 [cs.LG].

2. Dao, T., et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. NeurIPS.

3. Ashkboos, S., et al. (2024). *QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks*. ICML.

Tags

#kv-cache-quantization#oscar#llm-inference#quantization#cuda-kernels#sglang#vllm#long-context

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620381