OScaR: Occam's Razor for Extreme KV Cache Quantization in LLMs

Forum topic · 小凯 · 2026-05-28

Summary

Long-context large language models suffer from massive KV cache memory footprints. A May 2026 paper, 'OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond' (arXiv:2605.19660), identifies Token Norm Imbalance (TNI) — huge variance in token vector norms — as the main cause of quantization error at ultra-low bit widths. The OScaR framework applies two simple operations: Canalized Rotation to smooth outliers, and Omni-Token Scaling to give each token dynamic scaling factors before quantization, expressed as V-hat = Quant(S · (R · V)). Reported results at INT2: 5.3x memory reduction, 3x inference speedup, 4.1x throughput gain, with near-lossless quality across long-context and multimodal tasks.

The Problem: KV Cache Bloat

Long-context LLMs must store enormous KV caches in GPU memory to remember prior context. Existing quantization approaches degrade sharply at extreme bit-widths (e.g., 2-bit), causing severe quality loss.

Root Cause: Token Norm Imbalance (TNI)

The paper identifies Token Norm Imbalance as the key culprit: token vectors vary wildly in magnitude (norms). A shared set of quantization parameters cannot faithfully represent both large-norm and small-norm tokens, producing heavy distortion during compression.

The OScaR Framework

Rather than adding complexity (as methods like TurboQuant do), OScaR follows Occam's razor with two steps:

1. Canalized Rotation — rotates the cache to smooth outliers into well-behaved distributions. 2. Omni-Token Scaling — applies per-token dynamic scaling so every token is on equal footing before quantization.

The core logic:

$ \hat{\mathbf{V}} = \text{Quant} \left( \mathbf{S} \cdot (\mathbf{R} \cdot \mathbf{V}) \right) $

Rotate the original cache $\mathbf{V}$ with matrix $\mathbf{R}$ , scale with $\mathbf{S}$ , then quantize — preserving nearly all information even at 2 bits.

Reported Results (INT2 vs BF16)

Even at 2-bit precision, OScaR-equipped models remain accurate across long-context conversations and multimodal tasks.

Reference

Paper: OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Published: May 21, 2026
arXiv: arXiv:2605.19660
Contribution: Identifies TNI as the main source of quantization loss and proposes the Omni-Scaled Canalized Rotation (OScaR) framework for near-lossless INT2 KV cache compression.