OScaR: Occam's Razor for Extreme KV Cache Quantization in LLMs

Forum topic · 小凯 · 2026-05-28

Summary

OScaR is a framework for extreme KV cache quantization in large language models, addressing the memory bottlenecks of long-context inference. The paper identifies Token Norm Imbalance (TNI)—the large variance in vector norms across tokens—as the root cause of quantization error at very low bit-widths. Instead of using shared quantization parameters that crush small-norm tokens, OScaR applies two simple steps: Canalized Rotation, which suppresses outliers by rotating the cache, and Omni-Token Scaling, which applies per-token dynamic scaling so all tokens are quantized uniformly. The pipeline is expressed as V̂ = Quant(S·(R·V)). Reported results show INT2 quantization shrinks KV cache memory by 5.3x, accelerates inference by 3x, boosts throughput by 4.1x, and preserves model quality nearly losslessly on long-context and multimodal tasks. Paper: arXiv:2605.19660, released May 21, 2026.

Introduction

Modern long-context LLMs face a common pain point: the KV cache's GPU memory footprint. Caching "memory fragments" for every token bloats memory, and memory bandwidth becomes the bottleneck—loading more slows everything down. Naive quantization helps, but pushing to 2 bits causes severe degradation.

Root Cause: Token Norm Imbalance (TNI)

Previous quantization approaches implicitly assumed tokens have similar magnitudes. In reality, token vector norms vary dramatically. Applying a single shared quantization parameter wipes out small-norm tokens and misquantizes large ones, causing heavy distortion.

> Key concept: Token Norm Imbalance (TNI) — the huge spread of token vector norms within a sequence makes a unified quantization standard destructive.

The OScaR Framework

The OScaR framework (Omni-Scaled Canalized Rotation) takes a deliberately simple, two-step approach:

1. Canalized Rotation: rotates the cache to smooth out erratic outliers. 2. Omni-Token Scaling: per-token dynamic scaling so every token is equally well-suited to the quantizer.

The core logic is captured by:

$ \hat{\mathbf{V}} = \text{Quant} \left( \mathbf{S} \cdot (\mathbf{R} \cdot \mathbf{V}) \right) $

That is: rotate the raw cache ( $\mathbf{V}$ ) via rotation matrix ( $\mathbf{R}$ ), apply the omni-token scaling factor ( $\mathbf{S}$ ), then quantize.

Reported Results

Even at an aggressive 2-bit (INT2) setting, models equipped with OScaR remain coherent and robust across long conversations and multimodal benchmarks.

Reference

Title: OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Released: May 21, 2026
arXiv: arXiv:2605.19660
Problem: Excessive KV cache memory in long-context LLM inference, plus severe accuracy loss from extreme low-bit quantization.
Contribution: Identifies TNI as the main source of quantization loss and proposes the Omni-Scaled Canalized Rotation (OScaR) framework for near-lossless INT2 compression.