TurboQuant: How Polar Coordinate Quantization Shrinks LLM KV Cache Memory by 6x
*Translation and analysis of a zhichai.net forum post on Google's TurboQuant (arXiv:2504.19874).*
> Imagine reading a 1,000-page book and copying all previous pages each time you turn one. That sounds absurd — but large language models effectively do something similar every day, and they're running out of room.
1. The Problem: KV Cache "Obesity"
When you type into ChatGPT, every token is converted into vectors. Generating a reply requires attention: each new token must "look back" at all previous tokens. To avoid recomputing, models store all prior Key and Value results in the KV cache — like scratch paper for a long math problem.
For models with 128K context windows, the KV cache can consume tens of GB of VRAM. It only grows as the conversation continues, until hardware limits are hit.
2. The Quantization Dilemma
Quantization reduces bits per number (32-bit → 16 → 8 → 4). But scalar quantization requires per-block quantization constants (scale and zero-point), costing 1–2 extra bits per block. At 128K tokens × dozens of heads × hundreds of dimensions, this overhead is huge — like compressing a book into shorthand but attaching a decoder manual to every page.
3. The Polar Coordinate Insight
TurboQuant's PolarQuant converts vectors from Cartesian to polar coordinates (radius + angles) after a random rotation. Key insight: after random rotation, the angles of high-dimensional vectors follow a Beta distribution — tight and predictable. Like darts clustered in one region of a dartboard, the range is known in advance, so:
- Radius stored at full/high precision
- Angles quantized with few bits (2–3 bits)
- No runtime normalization or stored quantization constants needed
- Attention logits computation: up to 8x faster
- KV cache memory: at least 6x less
- Paper: *TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate* (arXiv:2504.19874)
- Authors: Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni (Google Research / Google DeepMind / NYU)
- Related paper: *PolarQuant: Quantizing KV Caches with Polar Transformation* (arXiv:2502.02617)
- Venues: ICLR 2026, AISTATS 2026
On Llama-3.1-8B, PolarQuant achieves over 4x compression at 3 bits per angle.
4. QJL: Fixing a Subtle Bias
MSE-optimized quantization introduces systematic bias in inner-product estimates — the basis of attention scores. QJL (Quantized Johnson-Lindenstrauss) fixes this:
1. Quantize with PolarQuant 2. Compute the residual (original minus reconstruction) 3. Apply a 1-bit random projection to the residual (store only the sign)
Mathematically, this yields an unbiased inner-product estimator — a simple check that cancels the bias, like calibrating a ruler that reads short.
5. The Full Pipeline
1. Random rotation — multiply by a random orthogonal matrix to make coordinates independent with predictable distributions 2. PolarQuant — convert to polar coordinates, quantize radius and angles 3. QJL residual correction (optional, for inner-product fidelity) — 1-bit sign of projected residual 4. Attention computation — reconstruct K/V, combine with QJL signs for unbiased inner products
The whole process is online, requiring no retraining or pre-scanned data.
6. Results
Needle-in-a-Haystack (finding a hidden sentence in a long document):
| Method | Score | |---|---| | SnapKV | 0.858 | | PyramidKV | 0.895 | | KIVI | 0.981 | | PolarQuant | 0.995 | | TurboQuant (4x) | 0.997 | | Full-precision baseline | 0.997 |
LongBench: On Llama-3.1-8B-Instruct, TurboQuant at 3.5 bits matches the full-precision baseline; at 2.5 bits, slight degradation but still beats other compression methods. Similar leading results on Mistral-7B-Instruct.
Performance (NVIDIA H100, 4-bit vs 32-bit baseline):
7. Vector Search
TurboQuant also applies to vector search (RAG, recommendation, retrieval). On the GloVe dataset (200-dim vectors), it beats Product Quantization and RabbiQ on Top-K recall, without large codebooks or dataset-specific tuning, with near-zero indexing time thanks to being data-oblivious.
8. Theoretical Guarantees
TurboQuant's distortion rate is proven to be within a constant factor (~2.7) of Shannon's information-theoretic lower bound — i.e., near-optimal. Its performance is bounded and predictable, so no edge-case collapses.
9. Industry Reaction
Cloudflare CEO Matthew Prince called it Google's "DeepSeek moment" — software innovation drastically changing hardware requirements. Memory supplier stocks dipped on release day (though Jevons paradox suggests such fears may be premature). Feature requests for TurboQuant support have appeared in vLLM's GitHub issues.
10. Looking Forward
TurboQuant embodies a trend: AI efficiency gains come not just from bigger models and hardware, but smarter algorithms. It suggests a future of 70B models running locally, enterprise AI at a tenth of the cost, and millisecond vector search at billion scale — all training-free, purely through algorithmic ingenuity bridging information theory, high-dimensional geometry, and GPU engineering.