TurboQuant: How Polar Coordinate Quantization Shrinks LLM KV Cache by 6x

*Translation of a Chinese forum post explaining Google's TurboQuant.*

> Imagine reading a 1,000-page book and copying everything before it each time you turn a page. That is essentially what large language models do with their KV Cache—and they are running out of room.

The Problem: KV Cache "Obesity"

When a model generates text, attention requires re-reading every previous token's key/value vectors. To avoid recomputing them, models store them in a KV Cache. For a model with a 128K context window, this cache can occupy tens of GB of GPU memory and only grows as the conversation continues.

Quantization and Its Hidden Cost

Quantization reduces the bits per number (e.g., 32-bit floats to 4-bit integers). But traditional scalar quantization must store quantization constants (scale, zero-point) per block—typically 1–2 extra bits per value. Across 128K tokens, dozens of heads, and hundreds of dimensions, this overhead eats much of the savings—like compressing a book but attaching a decoder manual to every page.

The Polar Coordinate Insight: PolarQuant

TurboQuant's core technique, PolarQuant, converts vectors from Cartesian to polar coordinates: one radius plus d−1 angles. After a random rotation, high-dimensional vectors' angles concentrate in a predictable range, following a Beta distribution. Because the range is known in advance, no runtime normalization or stored quantization constants are needed.

Result: on Llama-3.1-8B, PolarQuant compresses the KV cache by over 4x using only ~3 bits per angle.

QJL: Fixing a Subtle Bias

Optimizing quantization for MSE introduces systematic bias in inner-product estimates, which matters because attention is exactly query-key inner products. TurboQuant's second technique, QJL (Quantized Johnson-Lindenstrauss):

1. Quantizes with PolarQuant (MSE-optimized) 2. Computes the quantization residual 3. Applies a 1-bit random projection of the residual (store the sign)

This tiny correction yields an unbiased inner-product estimator when the (unquantized) query is dotted against reconstructed keys.

The Full Pipeline

1. Random rotation: multiply by a random orthogonal matrix 2. PolarQuant: convert to polar coordinates; quantize radius and angles with no extra constants 3. QJL residual correction (optional): 1-bit sign of projected residuals 4. Attention: reconstruct keys/values, optionally apply unbiased correction

Everything runs online at inference time—no retraining or fine-tuning required.

Results

Needle-in-a-Haystack:

• SnapKV: 0.858; PyramidKV: 0.895; KIVI: 0.981; PolarQuant: 0.995
• TurboQuant (4x compression): 0.997 — matching the full-precision baseline (0.997)

LongBench (Llama-3.1-8B-Instruct): at 3.5 bits, performance matches full precision; at 2.5 bits, slight degradation but still ahead of other compression methods. Similar results on Mistral-7B-Instruct.

Speed/memory on NVIDIA H100 (4-bit vs 32-bit baseline):

• Up to 8x faster attention logit computation
• At least 6x less KV cache memory

Vector Search

TurboQuant also applies to vector search (RAG, retrieval, recommendations). On GloVe 200-dim vectors, it beats Product Quantization and RabbiQ in Top-K recall without large codebooks or dataset-specific tuning, with near-zero indexing time since it is data-oblivious.

Theory

The authors prove TurboQuant's distortion rate is within a constant factor (~2.7) of Shannon's information-theoretic lower bound (distortion-rate function), i.e., near-optimal in a rigorous sense.

Industry Reaction

Cloudflare CEO Matthew Prince called it Google's "DeepSeek moment"—software innovation dramatically changing hardware requirements. Memory vendor stocks dipped on announcement day, though Jevons Paradox suggests efficiency gains often boost total demand. Support requests have already appeared in the vLLM GitHub repository.

Takeaway

TurboQuant works because it changes the coordinate system rather than squeezing each number harder—it chooses smarter representations instead of more effort. It demonstrates that training-free algorithmic design can still deliver large efficiency wins: smaller KV caches, longer contexts on the same hardware, and cheaper AI services.

References

• Paper: *TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate* (arXiv:2504.19874)
• Authors: Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni (Google Research / Google DeepMind / NYU)
• Related: *PolarQuant: Quantizing KV Caches with Polar Transformation* (arXiv:2502.02617)
• Venues: ICLR 2026, AISTATS 2026