TurboQuant+ Deep Dive: Ex-Google Engineer Ships Open-Source KV Cache Compression in 7 Days
*An English translation of a Chinese tech forum deep dive on the turboquant_plus project. English translation of zhichai.net post dated April 5, 2026.*
Introduction: An Absurd Opening
On March 24, 2026, Google Research published a paper titled *TurboQuant: Redefining AI Efficiency with Extreme Compression*. The claim was explosive: KV cache compressed to 3 bits, memory usage cut to one-sixth, 8x speedup, and supposedly "zero accuracy loss."
Within 48 hours, memory chip stocks tumbled. But Google released no code.
Tom Turney — who had just left Google after 13 years and listed himself as an "independent researcher" — saw the paper and thought: "If this works, it should run on every MacBook."
Three days later, he open-sourced turboquant_plus. Six days in, it had 1,274 stars. By day seven, you could run a 104B-parameter model on an M5 Max with 128K context and less memory than before.
Turney noted on Twitter that "I" really meant him working with Claude Code and Codex — his role was mostly "steering and babysitting."
This isn't a story about whether AI will replace programmers. It's about how much one person plus AI can accomplish against conventional assumptions.
Part 1: The KV Cache — LLMs' Memory Burden
Transformer models store computed Key and Value vectors (the KV cache) so attention doesn't have to recompute them. The cache grows linearly with context length: 100K tokens can eat tens of GB of VRAM. With million-token contexts now standard, the KV cache is the biggest inference bottleneck — not compute, but memory capacity and bandwidth.
Traditional mitigation is quantization (16-bit floats down to 8 or 4 bits), but aggressive quantization degrades output quality. Google's TurboQuant claimed to break this trade-off: 3 bits, 6x compression, zero loss.
Part 2: PolarQuant's Intuition — Direction Matters More Than Magnitude
TurboQuant's core is PolarQuant, built on a clean geometric insight: in attention, a vector's direction matters far more than its magnitude.
Attention is dot products, and dot product = direction similarity × magnitude product. PolarQuant converts Cartesian coordinates (x, y) to polar coordinates (angle θ, radius r), then compresses the radius aggressively while carefully protecting the angle.
They also apply a Walsh-Hadamard transform — a rotation that "spreads out" per-dimension values into a more Gaussian-like distribution, reducing quantization error.
Part 3: Day 1 Marathon — 90 Commits and a One-Line include Curse
Turney didn't build a throwaway PyTorch demo; he integrated the algorithm into llama.cpp.
Day 1 was a 90-commit marathon. When the first Metal GPU kernel ran, it hit 2.4 tokens/second (vs an expected ~85.5 on M5 Max). After hours of debugging, the culprit was a single include line that made the Metal compiler silently fall back to CPU — no error, just an apparently-GPU program actually spinning on CPU.
After the fix: 51.4 tok/s. Then the next blow: PPL = 165.6 (normal is 6–8). The model was producing complete gibberish. As he later quipped in the docs: "The speed benchmark measures how fast the model talks nonsense."
Day 1 ended with speed but no quality — but he'd stepped in the deepest pit and knew the problem was implementation, not the algorithm.
Part 4: 36 Hours to Working — From Gibberish to Poetry
Days 2–3 focused on the "gibberish." PolarQuant's rotation steps demand high numerical precision: float64 in Python was fine, but fp16 on GPU accumulated fatal rounding errors. He walked a tightrope between speed and correctness.
36 hours in, the first end-to-end test passed:
- Compression: 4.6x (turbo3)
- PPL: 6.176 vs the q8_0 baseline of 6.111 (+1.06%)
- Speed: near q8_0 prefill speeds
- 22.8% decode speedup at 32K context
- Zero perplexity change (verified across 50 chunks, CI ±0.021)
- Works across formats — q8_0, q4_0, not just TurboQuant
- 511+ Python tests, 100% coverage
- C port integrated into llama.cpp
- Metal GPU kernels supporting Apple Silicon M1–M5
- Community validation: CUDA (RTX 3080/3090/4090/5090), AMD (RX 9070 XT)
- Speed: 739 → 2747 tok/s (3.7x), parity with q8_0
- Tom Turney's GitHub: https://github.com/TheTom/turboquant_plus
- turboquant_plus project: https://github.com/TheTom/turboquant_plus
- llama.cpp fork: https://github.com/TheTom/llama-cpp-turboquant
- TurboQuant paper: arXiv:2504.19874 (ICLR 2026)
- PolarQuant paper: arXiv:2502.02617 (AISTATS 2026)
- Google Research Blog: TurboQuant: Redefining AI Efficiency
Part 5: Beyond the Paper — Three Unexpected Findings
Finding 1: V compression is "free"
Keys decide "where to look" (routing); Values decide "what information to fetch." Bad routing breaks everything downstream; slightly blurry Values are tolerable. Turney implemented asymmetric K/V compression: Keys at q8_0 (8-bit), Values at turbo3 (3-bit).
Finding 2: All quality loss comes from K compression
Further experiments confirmed: compressing Keys at low precision derails attention routing, but Values can be pushed to even 2 bits with almost no quality change. This explains why some models thrive with TurboQuant while others collapse — it depends on each model's sensitivity to Key precision.
Finding 3: Boundary layers are fragile
The first and last layers of a Transformer are most quantization-sensitive. His Boundary V scheme uses q8_0 for the first 2 and last 2 layers' Value caches and turbo2 for the middle: 15 lines of code, no speed loss, recovers 37–91% of quality loss.
Part 6: Sparse V Dequant — A 22.8% Speedup Gift
Turney's biggest original contribution: at long contexts (e.g., 32K tokens), attention distributions are extremely sparse — most positions have near-zero weights (< 1e-6). Sparse V Dequant uses attention weights as a gate to skip dequantizing low-weight Value vectors during decoding.
Results:
Needle-in-a-Haystack results improved from 7/9 to 9/9 single-needle retrieval with Sparse V enabled; multi-key retrieval hit 100% at 32K. He speculates that skipping low-weight quantization noise acts as denoising.
Part 7: The 7-Day Timeline
| Time | Milestone | |------|-----------| | Day 1 | Python prototype + llama.cpp integration, 90 commits, Metal shader pitfalls, PPL 165.6 nightmare | | Day 2–3 | Precision fixes, 141-line core stable, 500+ unit tests, 100% coverage | | Day 3–5 | C port + Metal GPU kernels, end-to-end run, speed optimization begins | | Day 5–7 | Extreme optimization: fp16 half-precision, half4 vectorized butterfly ops, graph-side rotation, block-32 storage layout |
Final results:
Part 8: Performance Panorama
Compression vs Quality
| Config | Bits/value | Compression | PPL (wikitext-2) | vs q8_0 | |--------|-----------|-------------|------------------|---------| | f16 | 16.0 | 1.0x | 6.121 | -0.16% | | q8_0 | 8.5 | 1.9x | 6.111 | baseline | | turbo4 | 4.25 | 3.8x | 6.125 | +0.23% | | turbo3 | 3.5 | 4.6x | 6.176 | +1.06% | | turbo2 | 2.5 | 6.4x | 6.507 | +6.48% |
Large-Model Stress Tests (M5 Max 128GB)
| Model | Params | Weights | KV config | PPL | Max context | NIAH | |-------|--------|---------|-----------|-----|-------------|------| | Llama-3.1-70B | 70B | Q4_K_M | turbo4/turbo4 | 3.461 | 48K | 30/30 | | Command-R+ | 104B | Q4_K_M | turbo3/turbo3 | 6.415 | 128K | 10/10 |
A 104B model with 128K context, on a MacBook — unthinkable a month earlier.
Sparse V Magic (Qwen3.5-35B-A3B MoE, 32K context)
| Config | Short text | 32K context | vs q8_0 | |--------|-----------|-------------|---------| | q8_0 | 85.71 tok/s | 1173.91 tok/s | baseline | | turbo3 | 76.84 tok/s | 1141.74 tok/s | 0.90x | | turbo3 + Sparse V | ~76 tok/s | ~1400 tok/s | ~1.19x |
With Sparse V, long-context decoding surpasses q8_0.
Part 9: Implications — One Person + AI = ?
AI is a lever, not a replacement
Turney didn't have Claude Code "write code for him." He read the paper, grasped the math, designed the architecture, diagnosed the include bug, and judged whether high PPL meant a precision or algorithm problem. Claude Code was his executor and sparring partner.
Small teams ship; big orgs publish papers
Google has thousands of engineers, but TurboQuant's code (if ever released) would wait months for reviews, compliance, and politics. One person did it in 7 days. AI-assisted coding compresses "implement an algorithm" from weeks to days — amplifying the small-team speed advantage tenfold.
The power of open source
From Day 3, the community poured in: CUDA validation on RTX 4090 and RTX 3090, independent Boundary V verification on NanoGPT, AMD RX 9070 XT support from community members. Six days, 174 commits, 1,274 stars, 30+ testers across M1/M2/M3/M5 Macs, NVIDIA, and AMD.
"Beyond the paper" becomes the norm
Turney didn't just reproduce the paper — Sparse V, Asymmetric K/V, Boundary V, and Temporal Decay are all his own. When implementation cost collapses, every engineer can be a researcher. Papers become starting points, not endpoints.
Epilogue
This isn't really about a KV cache compression algorithm. It's about one person beating a big company to shipping, about AI as an engineering lever that lets individuals breach organizational boundaries, and about open source turning a paper into a production tool for 104B models on laptops in 7 days.
From his README:
> "If individual modules prove useful and stable, the goal is to progressively upstream them into llama.cpp as small, reviewable patches."
Postscript: What That include Line Taught Me
Engineering isn't about big breakthroughs — it's about small details. A wrong include, an under-precise intermediate variable, a missed edge case: these separate "runs" from "runs well." Tom proved in 7 days that AI can write your code, but judging whether the code is right, why it's wrong, and how to fix it still takes a human.
At least for now.