TurboQuant+ Deep Dive: Ex-Google Engineer Reimplements Google's Extreme KV-Cache Compression in 7 Days
> Style reference: Paul Graham's essay style (concise and forceful) + Feynman's physical intuition (explaining complex ideas like neighborhood chat)
---
Prologue: An Absurd Opening
On March 24, 2026, Google Research published a paper with an unremarkable title: *TurboQuant: Redefining AI Efficiency with Extreme Compression*. But the content hit like a depth charge — they compressed the KV cache of large-model inference to 3 bits, cut memory usage to one-sixth, achieved up to 8x speedups, and claimed "zero accuracy loss."
Within 48 hours, memory-chip stocks worldwide lost billions. Micron and SK Hynix shares fell like kites with cut strings.
But Google released no code.
"Here's the paper, the principles are clear, play with it yourselves."
When Tom Turney saw the news, he was sitting in his apartment. He had just left Google after 13 years; his LinkedIn read "independent researcher." Staring at the paper, he had one thought:
"If this thing is real, it should run on every MacBook."
Three days later, he open-sourced turboquant_plus. Six days in, the project had 1,274 stars. By day seven, you could run a 104B-parameter model on an M5 Max with 128K context and less memory than before.
And it was all done by "him" and Claude Code — Tom noted on Twitter: "When I say 'I', it's actually with Claude Code and Codex; the main work was a lot of steering and babysitting."
This is not a story about "will AI replace programmers." It's a story about "how many conventional assumptions can one person plus AI break."
---
Part 1: KV Cache — the LLM's Memory Burden
To understand why TurboQuant matters, you first need to understand the memory black hole of LLM inference.
Imagine reading a thick book: to understand each page, you must remember everything before it. Transformers work the same way — attention "reviews" all previous tokens. To avoid recomputation, the model stores the computed "Key" and "Value" vectors: the KV cache.
The problem: this cache grows linearly with context length. 1,000 tokens? Small cache. 100,000 tokens? Tens of GB of VRAM. Flagship models now support million-token contexts, and the KV cache has become the biggest bottleneck — not compute, but memory capacity and bandwidth.
The traditional fix is quantization: compress 16-bit floats to 8, 4, or fewer bits. But quantization has a cost — compress too hard and the model starts "talking nonsense."
Google's TurboQuant claims to break the trade-off curve: 3 bits, 6x compression, zero loss.
---
Part 2: PolarQuant's Intuition — Why Direction Matters More Than Magnitude
TurboQuant's core is an algorithm called PolarQuant. Tom saw a beautiful mathematical intuition in the paper:
In attention computation, a vector's direction matters far more than its magnitude.
Think of navigation: the target is 5 km east. "East" is direction, "5 km" is distance. Change the distance to 4.8 km and you'll probably still find it; get the direction wrong by 30 degrees and you're completely lost.
Attention is essentially dot products between vectors, and dot product = directional similarity × magnitude product.
PolarQuant converts Cartesian coordinates (x, y) into polar coordinates (angle θ, radius r), then compresses the radius aggressively (it matters less) while carefully protecting the angle.
Even better, they add a Walsh-Hadamard transform — a rotation that "scrambles" vector dimensions, making unevenly distributed values nearly Gaussian. Uniform values mean smaller quantization error.
Reading this, Tom's eyes lit up. This isn't just engineering — it's the triumph of geometric intuition.
---
Part 3: Day 1 Marathon — 90 Commits and the Curse of One Include
Tom decided to implement the algorithm from scratch — not a hasty PyTorch demo, but integrated into llama.cpp, the open-source inference engine that runs LLMs on laptops.
Day 1 was insane: 90 commits, morning to midnight. He called it a "marathon day."
By evening the first Metal GPU kernel ran. Tom pressed enter with high hopes. Result: 2.4 tokens/second.
An M5 Max should hit 85.5 tok/s. 2.4? Slower than CPU.
After hours of debugging, the culprit was a single include line that made the Metal compiler silently fall back to CPU mode — no error, the program "looked" like it was running on GPU while softly spinning its wheels.
After the fix: 51.4 tok/s. But then another blow: PPL = 165.6.
Perplexity measures output quality; normal values are around 6–8. 165.6 meant the model was speaking gibberish.
Tom later joked in the docs: "Speed benchmarks measure how fast the model talks nonsense."
Day 1 ended with speed but no quality. Yet he had cleared the deepest pit — he knew the problem was implementation detail, not the algorithm.
---
Part 4: 36 Hours to Working — From Gibberish to Poetry
Days 2 and 3 focused on fixing the gibberish.
He found that PolarQuant's rotation step demands extreme numerical precision. float64 in Python was fine, but fp16 on GPU accumulated errors until the model collapsed. He spent a long time tuning intermediate precision, walking a tightrope between speed and correctness.
After 36 hours, the first end-to-end test passed:
- Compression: 4.6x (turbo3)
- PPL: 6.176 vs. the q8_0 baseline of 6.111 — only +1.06%
- Speed: close to q8_0 prefill speed
- 22.8% decode speedup at 32K context
- Zero perplexity change (validated across 50 chunks, CI ±0.021)
- Works not just with TurboQuant but also q8_0 and q4_0
- 511+ Python tests, 100% coverage
- C port integrated into llama.cpp
- Metal GPU kernels for Apple Silicon M1–M5
- Community validation: CUDA (RTX 3080/3090/4090/5090), AMD (RX 9070 XT)
- Speed: from an initial 739 tok/s optimized to 2747 tok/s — 3.7x improvement, on par with q8_0
- Read the paper and understood the core mathematical intuition
- Designed the architecture — where to validate in Python, where to implement in C
- Diagnosed problems (that damned include)
- Judged direction (is high PPL a precision issue or an algorithm issue?)
- Tom Turney's GitHub: https://github.com/TheTom/turboquant_plus
- turboquant_plus project: https://github.com/TheTom/turboquant_plus
- llama.cpp fork: https://github.com/TheTom/llama-cpp-turboquant
- TurboQuant paper: arXiv:2504.19874 (ICLR 2026)
- PolarQuant paper: arXiv:2502.02617 (AISTATS 2026)
- Google Research Blog: TurboQuant: Redefining AI Efficiency
The model stopped talking nonsense. It wrote poems, code, and answers — nearly indistinguishable from full precision.
---
Part 5: Beyond the Paper — Sparse V and Three Unexpected Findings
Tom didn't stop at "reproducing the paper." Integrating into llama.cpp, he discovered three phenomena the original paper never mentioned:
Finding 1: V compression is "free"
TurboQuant compresses Key and Value caches symmetrically. But Tom found: Value vectors can be compressed far more aggressively with almost no quality loss.
His explanation: in attention, Keys decide "where to look" (routing); Values decide "what information to fetch." Wrong routing ruins everything downstream, but slightly fuzzy Values are fine.
So he implemented asymmetric K/V compression: Keys at q8_0 (8-bit), Values at turbo3 (3-bit) — rescuing quantization-sensitive models while keeping most memory gains.
Finding 2: All quality loss comes from K compression
Experiments confirmed the intuition: low-precision Keys scramble attention routing, and the model "wanders off." Compress only Values — even to 2 bits — and quality barely changes. This explains why some models thrive with TurboQuant while others collapse — it depends on each model's sensitivity to Key precision.
Finding 3: Boundary layers are especially fragile
Tom found a Transformer's first and last few layers are most quantization-sensitive. Middle layers can be compressed freely; boundary layers need "gentle treatment."
He implemented Boundary V: first 2 and last 2 layers' Value caches at q8_0, middle layers at turbo2. Fifteen lines of code, zero speed loss, recovering 37–91% of quality loss.
---
Part 6: Sparse V Dequant — a 22.8% Speedup Gift
Tom's biggest original contribution is Sparse V Dequant.
The idea came from observing attention weights: with long contexts (e.g., 32K tokens), each new token's attention distribution is extremely sparse — the vast majority of positions have near-zero weight (< 1e-6).
If those positions contribute almost nothing, why spend time dequantizing them?
Sparse V uses attention weights as a "gate" during decode, skipping dequantization of low-weight Value vectors.
The results were stunning:
This is a format-agnostic optimization. Tom: "This isn't a TurboQuant trick — it's an inherent property of attention."
More surprisingly, Needle-in-a-Haystack results: with Sparse V enabled, single-needle retrieval went from 7/9 to 9/9 (100%); multi-key retrieval hit 100% at 32K. Tom speculates that skipping low-weight quantization noise acts as a kind of "denoising."
---
Part 7: The 7-Day Timeline — a Race Against Time
| Time | Milestone | |------|-----------| | Day 1 | Python prototype + llama.cpp integration, 90 commits, Metal shader pitfalls, the PPL-165.6 nightmare | | Day 2–3 | Fixed precision issues; 141-line stable core; 500+ unit tests, 100% code coverage | | Day 3–5 | C port + Metal GPU kernels; end-to-end run; speed optimization begins | | Day 5–7 | Extreme optimization: fp16 half-precision, half4 vectorized butterfly ops, graph-side rotations, block-32 storage layout |
Final results:
---
Part 8: Performance Panorama — the Story Behind the Numbers
Compression and Quality
| Config | Bits/value | Compression | PPL (wikitext-2) | vs. q8_0 | |--------|-----------|-------------|------------------|----------| | f16 | 16.0 | 1.0x | 6.121 | -0.16% | | q8_0 | 8.5 | 1.9x | 6.111 | baseline | | turbo4 | 4.25 | 3.8x | 6.125 | +0.23% | | turbo3 | 3.5 | 4.6x | 6.176 | +1.06% | | turbo2 | 2.5 | 6.4x | 6.507 | +6.48% |
turbo4's quality is essentially on par with q8_0 (within error margins). A victory for PolarQuant — geometric intuition beating brute-force quantization.
Large-Model Stress Tests (M5 Max 128GB)
| Model | Params | Weight precision | KV config | PPL | Max context | NIAH | |-------|--------|------------------|-----------|-----|-------------|------| | Llama-3.1-70B | 70B | Q4_K_M | turbo4/turbo4 | 3.461 | 48K | 30/30 | | Command-R+ | 104B | Q4_K_M | turbo3/turbo3 | 6.415 | 128K | 10/10 |
A 104B model, 128K context, on a MacBook. A month ago, that was fantasy.
The Sparse V Magic
Qwen3.5-35B-A3B (MoE) decode speed at 32K context:
| Config | Short text | 32K context | vs. q8_0 | |--------|-----------|-------------|----------| | q8_0 | 85.71 tok/s | 1173.91 tok/s | baseline | | turbo3 | 76.84 tok/s | 1141.74 tok/s | 0.90x | | turbo3 + Sparse V | ~76 tok/s | ~1400 tok/s | ~1.19x |
With Sparse V, long-context decode speed overtakes q8_0.
---
Part 9: Meaning — One Person + AI = ?
The most moving thing about Tom Turney's project isn't the technical detail — it's the new engineering paradigm it demonstrates.
Insight 1: AI is a lever, not a replacement
Tom didn't let Claude Code "write code for him." He:
Claude Code was his executor and thinking partner. He asked questions, AI generated candidate code, he verified, adjusted, asked again.
This is human-machine chemical collaboration — human intuition + AI generation = output beyond either alone.
Insight 2: Big labs publish papers; small teams ship
Google has thousands of engineers, but TurboQuant's code — if they ever release it — might wait months for internal review, compliance, and politics.
Tom, alone: 7 days.
This isn't about Google's engineers being inadequate. It's that a small team's iteration speed can crush a large organization's bureaucratic inertia. When AI-assisted programming compresses "implement an algorithm" from weeks to days, the advantage of moving fast is amplified tenfold.
Insight 3: The power of open source
turboquant_plus isn't Tom's work alone. From Day 3, the community poured in: @sztlink validated the CUDA path on RTX 4090; @HyperionMS2040 got it running on RTX 3090; @Corianas_ independently verified Boundary V on NanoGPT; AMD RX 9070 XT support came from a community member.6 days, 174 commits, 1,274 stars, 30+ testers across M1/M2/M3/M5 Macs, NVIDIA, and AMD.
This is the terrifying power of open source — once the seed is planted, it grows exponentially.
Insight 4: "Beyond the paper" becomes the norm
Tom didn't just reproduce the paper — he delivered Sparse V, Asymmetric K/V, Boundary V, Temporal Decay, and other optimizations the original paper lacked.
This will only become more common. As implementation cost drops, every engineer can be a researcher — experimenting while implementing, discovering phenomena while benchmarking.
Papers are no longer the end point, but the starting point.
---
Epilogue: What Are We Really Talking About
We're not talking about a KV-cache compression algorithm.
We're talking about one person deciding to take on a giant — and winning.
We're talking about AI as an engineering lever that lets individuals break organizational boundaries.
We're talking about an open-source community turning a paper into a production tool that runs 104B models on laptops — in 7 days.
Tom Turney wrote in his README:
> "If individual modules prove useful and stable, the goal is to progressively upstream them into llama.cpp as small, reviewable patches."
A humble ambition. He doesn't want a private kingdom — he wants to push the whole ecosystem forward.
1,274 stars are the community's answer. And the developers running 128K contexts on M5 Max machines are the real protagonists of this story.
---
References
Postscript: What That Include Line Taught Me
After finishing this piece, I went back to Tom's Day 1 log. That include line — silently dropping Metal to CPU — reminded me of countless similar pits I've fallen into.
Engineering isn't about big breakthroughs; it's about small details.
A wrong include, an under-precise intermediate variable, a missed boundary condition — these are what separate "runs" from "runs well."
In 7 days, Tom proved that AI can help you write code, but judging whether the code is right, why it's wrong, and how to fix it — that still takes a human.
For now, at least.
---
*Written April 5, 2026, based on Tom Turney's turboquant_plus project documentation and community discussions.*