TurboQuant vs RotorQuant: The Truth About KV Cache Compression and Speed
Google's TurboQuant paper claims 5x+ KV cache compression for LLMs—turning a 100K-token context into 500K tokens on the same GPU. Yet in practice, many users found that **prompt processing got *slower* after compression: on Apple Silicon Metal GPUs, prefill ran at only 78% of Q8_0 quantization speed. Meanwhile, newer community methods—RotorQuant, IsoQuant, PlanarQuant—claimed 9–31x speedups. This article traces what's actually true.
Background: What is the KV cache?
LLMs generate tokens one at a time, so every transformer layer must store per-token K (key) and V (value) vectors for attention matching. For a 70B model (80 layers, 8 heads, 128 dims) at 32K context with FP16, the KV cache alone is ~10.4 GB; at 128K it would be 41.6 GB. This is the memory wall**: weights can be quantized offline, but the KV cache is *dynamic*—generated online per conversation—and quantization errors pass through softmax's exponential, accumulate, and can turn output into gibberish.
TurboQuant's insight
1. Random orthogonal rotation makes coordinates of a d-dimensional unit vector follow Beta(1/2, (d−1)/2) ≈ N(0, 1/d), i.e., approximately independent and identically distributed. Lloyd-Max scalar quantization then approaches the information-theoretic optimum (within ~2.7x constant). 2. QJL residual correction: MSE-optimal quantization shrinks vector norms; a 1-bit random projection of the residual makes inner-product estimates unbiased.
Theory is elegant. Practice was not.
The real bottleneck on Metal
TheTom (author of the llama.cpp TurboQuant implementation) documented the debugging: prefill speed degraded sharply with context length. Suspects like the O(d log d) Walsh-Hadamard Transform, custom WHT kernels, and group-32 rotations all failed. The actual culprit: Metal's dequant kernel read quantized data byte by byte, extracting 3-bit indices element-by-element—each SIMD thread re-reading the same byte, causing a high compute graph split count.
Fix: batch byte reads into uint32 loads and extract all indices in registers. Result: turbo3 stabilized at 98.7–99.5% of q8_0 speed on M5 Max. Meanwhile, on CPU (memory-bandwidth-bound), TurboQuant actually *beat* FP16: tq3_0 prompt processing at 20.1 tok/s vs 19.3 tok/s for FP16.
Community findings: simpler is better
MSE-only beats MSE+QJL
At least 6 independent teams (TheTom, Aaryan-Kapoor, arclabs001, scos-lab, tonbistudio, 0xSero) found dropping QJL improves quality. YATQ measurements (Top-1 token agreement):
| Bits | MSE-only | +QJL | Gap | |------|----------|------|-----| | 2-bit | 65.6% | 50.0% | −15.6% | | 3-bit | 71.0% | 61.2% | −9.8% | | 4-bit | 80.4% | 69.6% | −10.7% |
Why? QJL removes bias but increases variance. Softmax's exponential is convex, so by Jensen's inequality an unbiased inner-product estimate yields biased (systematically inflated) attention weights. Softmax tolerates a uniform bias but not noisy rankings—variance corrupts Top-K ordering. On GPT-2, a QJL-based method showed +300% perplexity vs +7.6% for MSE-only.
K/V asymmetric needs
Modern LLMs have wildly different K and V norms:
| Model | K mean norm | V mean norm | Ratio | |-------|-------------|-------------|-------| | GPT-2 (124M) | 11.8 | 2.0 | 6x | | Qwen2.5-7B | 274.0 | 2.6 | 106x | | Qwen2.5-1.5B | 778.6 | 4.3 | 182x |
Since quantization error scales with squared norm, K needs far more bits (4–5-bit, or mixed precision with 8-bit outlier channels) while V can use 2–3 bits.
Auditing RotorQuant's 9–31x claims
RotorQuant/PlanarQuant/IsoQuant use small 2D/3D rotations (Givens, Clifford rotors, SO(4) quaternions) instead of full d-dimensional WHT—O(d) instead of O(d log d), with 128 vs 16,384 parameters. On Llama 3.1 8B / RTX 5090:
| Config | Decode tok/s | Prefill tok/s | PPL | |--------|--------------|---------------|-----| | FP16 | 140 | 6,156 | 6.63 | | iso3/iso3 | 118 | 3,397 | 6.91 | | planar3/planar3 | 119 | 3,822 | 7.05 | | turbo3/turbo3 | 93 | 722 | 7.07 |
The "9–31x" headline compares PlanarQuant prefill against *unoptimized* TurboQuant prefill—not against baseline. Against FP16, PlanarQuant is actually 38% slower. Its real advantages are higher compression (~10.3x vs ~4.4x) and cheaper rotation, at the cost of larger quality loss (+5–6% PPL) and slower absolute speed. Its "deferred quantization" trick also keeps K in FP16 during prefill, so compression only pays off during long-context decode.
Hardware realities
- Metal: fixed byte-read issue recovers to 98–99% of q8_0, but hard to beat FP16; fused Flash Attention kernels still in progress.
- CUDA: decode ~15% below FP16 on RTX 5090—quantization saves bandwidth but high-end cards aren't bandwidth-limited at decode.
- CPU: the most successful platform—bandwidth-bound, so compression genuinely helps.
- TurboQuant: arXiv:2504.19874 (ICLR 2026)
- PolarQuant: arXiv:2502.02617 (AISTATS 2026)
- QJL: arXiv:2406.03482 (AAAI 2025)
- IsoQuant: arXiv:2603.28430
- TheTurboQuant+: https://github.com/TheTom/turboquant_plus
- CPU implementation: https://github.com/Aaryan-Kapoor/llama.cpp/tree/turboquant-tq3_0
- YATQ: https://github.com/arclabs001/YATQ
- RotorQuant: https://github.com/scrya-com/rotorquant
- turboquant-pytorch: https://github.com/tonbistudio/turboquant-pytorch
Honest recommendations
Use TurboQuant (MSE-only) when: running long contexts (128K+), memory-constrained hardware (24GB consumer GPUs, Apple Silicon), CPU inference, or quality-sensitive applications (only 1–2% PPL loss at 3-bit).
Avoid when: short contexts (4–8K), chasing maximum throughput, or latency-critical first-token response.
On RotorQuant family: promising direction, but the "9–31x speedup" marketing is misleading—it benchmarks against unoptimized implementations, not FP16. Prefer TurboQuant's MSE-only implementations today; watch the 2D/3D rotation family mature.
The theory is correct—but it's a map, not the territory. Only community debugging of Metal kernels, memory layouts, and PPL numbers made these theoretically perfect algorithms actually usable.