TurboQuant+ Deep Dive: Ex-Google Engineer Recreates Google's KV Cache Paper in 7 Days
> Written in the spirit of Paul Graham's essays (concise and forceful) plus Feynman-style physical intuition (explaining complex ideas like neighborhood small talk).
---
Prologue: An Absurd Opening
On March 24, 2026, Google Research published a paper with an unremarkable title: *TurboQuant: Redefining AI Efficiency with Extreme Compression*. But its content landed like a depth charge — they compressed LLM inference KV caches to 3 bits, cut memory to one-sixth, achieved 8x speedups, and claimed "zero accuracy loss."
Within 48 hours, memory chip stocks worldwide lost billions in market value. Micron and SK Hynix shares plummeted.
But Google released no code.
"Here's the paper. The principles are clear. Have fun."
When Tom Turney saw the news, he was sitting in his apartment. He had just left Google after 13 years; his LinkedIn said "independent researcher." Staring at the paper, he had one thought:
"If this is real, it should run on every MacBook."
Three days later, he open-sourced turboquant_plus. Six days in, the project had 1,274 stars. On day seven, you could run a 104B-parameter model on an M5 Max with 128K context and less memory than before.
And he did it *with* Claude Code — Tom noted on Twitter: "When I say 'I', it's actually together with Claude Code and Codex; my main job was lots of steering and babysitting."
This isn't a story about "will AI replace programmers." It's a story about how much one person plus AI can overturn conventional assumptions.
---
Part 1: The KV Cache — LLMs' Memory Burden
To understand why TurboQuant matters, you need to understand the memory black hole of LLM inference.
Imagine reading a thick book where you must remember every previous page to understand context. Transformers work this way — attention "looks back" at all previous tokens. To avoid recomputing, they store the computed Key and Value vectors: the KV cache.
The problem: this cache grows linearly with context length. A 1,000-token context needs a small cache; 100K tokens can eat tens of GB of VRAM. Flagship models now support million-token contexts, making the KV cache the biggest bottleneck — not compute, but memory capacity and bandwidth.
The traditional fix is quantization: compressing 16-bit floats to 8-bit, 4-bit, or lower. But quantization has costs — compress too hard and the model starts "talking nonsense."
Google's TurboQuant claimed to break through this trade-off curve: 3 bits, 6x compression, zero loss.
---
Part 2: PolarQuant's Intuition — Why Direction Beats Magnitude
TurboQuant's core is an algorithm called PolarQuant. Tom saw a beautiful mathematical intuition in the paper:
**In attention computation, vector *direction* matters far more than *magnitude*.
Think of navigation: your target is 5 km east. "East" is direction, "5 km" is distance. If the distance becomes 4.8 km, you'll still find the place; if the direction is off by 30 degrees, you're completely lost.
Attention fundamentally computes dot products, and dot product = directional similarity × magnitude product.
PolarQuant converts Cartesian coordinates (x, y) to polar coordinates (angle θ, radius r). Then it compresses the radius aggressively (it matters less) while carefully protecting the angle.
Better yet, they add a Walsh-Hadamard transform — a rotation that "scrambles" vector dimensions, making unevenly distributed values approximately Gaussian. Uniform distributions mean smaller quantization error.
When Tom read this section, his eyes lit up. This isn't just an engineering trick — it's a triumph of geometric intuition.
---
Part 3: The Day 1 Marathon — 90 Commits and the Curse of One include
Tom decided to implement the algorithm from scratch. Not a quick PyTorch demo, but integrated into llama.cpp — the open-source inference engine that runs LLMs on laptops.
Day 1 was a mad dash: 90 commits from morning to midnight — a "marathon day."
By evening, the first Metal GPU kernel was running. Tom hit enter full of hope. Result: 2.4 tokens/second.
An M5 Max should hit 85.5 tok/s. 2.4? Slower than CPU.
Hours of debugging revealed a single include line that made the Metal compiler silently fall back to CPU mode — no error, the program "looked" like it was running on GPU while actually spinning on CPU.
After the fix, speed jumped to 51.4 tok/s. But before he could celebrate, another blow: PPL = 165.6.
Perplexity measures output quality; normal values are around 6–8. 165.6 meant the model was spouting gibberish — not slightly wrong, complete nonsense.
Tom later joked in the docs: "Speed benchmarks measure how fast the model talks nonsense."
That was Day 1's ending: speed but no quality. But he'd crossed the deepest pit — he knew the problem was in implementation details, not the algorithm.
---
Part 4: Working in 36 Hours — From Gibberish to Poetry
Days 2 and 3 focused on fixing the "nonsense."
He found that PolarQuant's rotation step demands extremely high numerical precision. Computing in float64 in Python worked fine, but accumulating fp16 error on GPU broke the model. He spent extensive time tuning intermediate precision, walking a tightrope between speed and correctness.
After 36 hours, the first end-to-end test passed:
- Compression ratio: 4.6x (turbo3)
- PPL: 6.176, versus the q8_0 baseline of 6.111 — only 1.06% higher
- Speed: close to q8_0 prefill speeds
- 22.8% decode speedup at 32K context
- Zero perplexity change (validated over 50 chunks, CI ±0.021)
- Works not just with TurboQuant but also q8_0 and q4_0
- 511+ Python tests, 100% coverage
- C port integrated into llama.cpp
- Metal GPU kernels supporting Apple Silicon M1–M5
- Community validation: CUDA (RTX 3080/3090/4090/5090), AMD (RX 9070 XT)
- Speed: optimized from 739 tok/s to 2747 tok/s — a 3.7x improvement, on par with q8_0
- Read the paper and understood the core mathematical intuition
- Designed the architecture: where to validate in Python, where to implement in C
- Diagnosed problems (that damn
include) - Judged direction: why is PPL so high — precision or algorithm?
- @sztlink validated the CUDA path on an RTX 4090
- @HyperionMS2040 got it running on an RTX 3090
- @Corianas_ independently verified Boundary V on NanoGPT
- AMD RX 9070 XT support came from a community member
- Tom Turney's GitHub: https://github.com/TheTom/turboquant_plus
- turboquant_plus project: https://github.com/TheTom/turboquant_plus
- llama.cpp fork: https://github.com/TheTom/llama-cpp-turboquant
- TurboQuant paper: arXiv:2504.19874 (ICLR 2026)
- PolarQuant paper: arXiv:2502.02617 (AISTATS 2026)
- Google Research Blog: TurboQuant: Redefining AI Efficiency
The model stopped talking nonsense. It wrote poetry, code, and answers — nearly indistinguishable from full precision.
Tom pushed to GitHub and began writing optimized Metal GPU kernels.
---
Part 5: Beyond the Paper — Sparse V and Three Unexpected Findings
Tom didn't stop at "reproducing the paper." Integrating the algorithm into llama.cpp, he found three phenomena the original paper never mentioned:
Finding 1: V Compression Is "Free"
TurboQuant compresses Key and Value caches symmetrically. But Tom found: Value vectors can be compressed far more aggressively with almost no quality loss.**
His explanation: in attention, Keys decide *where to look* (routing), Values decide *what information to take*. Wrong routing ruins everything downstream; slightly blurry Values barely matter.
Based on this, he implemented asymmetric K/V compression: Keys at q8_0 (8-bit), Values at turbo3 (3-bit) — rescuing quantization-sensitive models while keeping most memory gains.
Finding 2: All Quality Loss Comes from K Compression
Further experiments confirmed the intuition. Low-precision Keys scramble attention routing and the model "loses focus." But compressing Values alone — even to 2 bits — barely changes quality.
This explains why some models do great with TurboQuant and others collapse — it depends on each model's sensitivity to Key precision.
Finding 3: Boundary Layers Are Especially Fragile
Tom also found the Transformer's first and last layers are most quantization-sensitive. Middle layers can be compressed freely; boundary layers need gentle treatment.
He implemented Boundary V: the first 2 and last 2 layers' Value caches use q8_0, middle layers use turbo2. Fifteen lines of code, no speed loss, recovering 37–91% of quality loss.
---
Part 6: Sparse V Dequant — A 22.8% Speedup Gift
Tom's biggest original contribution is Sparse V Dequant.
The idea came from observing attention weights: at long contexts (e.g., 32K tokens), a model's attention over previous tokens for each new token is extremely sparse — the vast majority of positions have near-zero attention weight (< 1e-6).
If these positions contribute almost nothing, why spend time dequantizing them?
Sparse V uses attention weights as a "gate" during decoding, skipping dequantization of low-weight Value vectors.
The results were striking:
This is a format-agnostic optimization. Tom said: "This isn't a TurboQuant trick; it's an inherent property of attention."
Even more surprising: Needle-in-a-Haystack results improved from 7/9 to 9/9 (100%) single-needle retrieval with Sparse V on; multi-key retrieval hit 100% at 32K context.
Tom speculates that skipping low-weight quantization noise may act as a "denoising" effect.
---
Part 7: The 7-Day Timeline — A Race Against Time
| Time | Milestone | |------|-----------| | Day 1 | Python prototype + llama.cpp integration, 90 commits, Metal shader pitfalls, the PPL 165.6 nightmare | | Day 2–3 | Fixed precision issues, 141-line core algorithm stabilized, 500+ unit tests, 100% code coverage | | Day 3–5 | C port + Metal GPU kernels, end-to-end run, optimization begins | | Day 5–7 | Extreme optimization: fp16 half-precision, half4 vectorized butterfly ops, graph-side rotations, block-32 storage layout |
Final results:
---
Part 8: Performance Panorama — The Stories Behind the Numbers
Compression and Quality
| Config | Bits/value | Compression | PPL (wikitext-2) | vs q8_0 | |--------|-----------|-------------|------------------|---------| | f16 | 16.0 | 1.0x | 6.121 | -0.16% | | q8_0 | 8.5 | 1.9x | 6.111 | baseline | | turbo4 | 4.25 | 3.8x | 6.125 | +0.23% | | turbo3 | 3.5 | 4.6x | 6.176 | +1.06% | | turbo2 | 2.5 | 6.4x | 6.507 | +6.48% |
turbo4's quality is essentially as good as q8_0 (within error margins). A victory for PolarQuant — geometric intuition beats brute-force quantization.
Large-Model Stress Tests (M5 Max 128GB)
| Model | Params | Weights | KV config | PPL | Max context | NIAH | |-------|--------|---------|-----------|-----|-------------|------| | Llama-3.1-70B | 70B | Q4_K_M | turbo4/turbo4 | 3.461 | 48K | 30/30 | | Command-R+ | 104B | Q4_K_M | turbo3/turbo3 | 6.415 | 128K | 10/10 |
A 104B model with 128K context, on a MacBook. A month ago, that was fantasy.
The Sparse V Magic
Qwen3.5-35B-A3B (MoE) decode speed at 32K context:
| Config | Short text | 32K context | vs q8_0 | |--------|-----------|-------------|---------| | q8_0 | 85.71 tok/s | 1173.91 tok/s | baseline | | turbo3 | 76.84 tok/s | 1141.74 tok/s | 0.90x | | turbo3 + Sparse V | ~76 tok/s | ~1400 tok/s | ~1.19x |
With Sparse V enabled, long-context decode speed overtakes q8_0.
---
Part 9: Meaning and Lessons — One Person + AI = ?
The most moving thing about Tom Turney's project isn't the technical detail — it's the new engineering paradigm it demonstrates.
Lesson 1: AI Is a Lever, Not a Replacement
Tom didn't have Claude Code "write code for him." He:
Claude Code was his executor and sparring partner. He posed questions, AI generated candidate code, he verified, adjusted, and asked again.
This is human-AI chemical collaboration — human intuition + AI generation = output beyond what either achieves alone.
Lesson 2: Big Companies Publish Papers; Small Teams Commercialize
Google has thousands of engineers, but TurboQuant's code — if they ever release it — might wait months through reviews, compliance checks, and internal politics.
Tom, alone: 7 days.
This doesn't mean Google's engineers aren't good. It means a small team's iteration speed can crush a large organization's bureaucratic inertia.
When AI-assisted coding compresses "implement an algorithm" from weeks to days, the advantage of moving fast is amplified tenfold.
Lesson 3: The Power of Open Source
turboquant_plus isn't Tom's work alone. From Day 3, the community poured in:
6 days, 174 commits, 1,274 stars, 30+ testers across M1/M2/M3/M5 Macs, NVIDIA, and AMD.
That's the terrifying power of open source — once the seed is planted, it grows exponentially.
Lesson 4: "Beyond the Paper" Becomes the Norm
Tom didn't just reproduce the paper; he delivered Sparse V, Asymmetric K/V, Boundary V, and Temporal Decay — optimizations absent from the original work.
This will become increasingly common. When implementation cost drops, every engineer can be a researcher — experimenting while implementing, discovering new phenomena while benchmarking.
Papers are no longer endpoints; they're starting points.
---
Epilogue: What Are We Really Talking About When We Talk About TurboQuant+
We're not talking about a KV cache compression algorithm.
We're talking about one person deciding to take on a giant — and winning.
We're talking about AI as an engineering lever that lets individuals break through organizational boundaries.
We're talking about an open-source community turning a paper into a production tool that runs 104B models on laptops — in 7 days.
Tom Turney wrote in his README:
> "If individual modules prove useful and stable, the goal is to progressively upstream them into llama.cpp as small, reviewable patches."
A humble ambition. He doesn't want a separate kingdom; he wants to push the whole ecosystem forward.
1,274 stars were the community's answer. And the developers running 128K contexts on M5 Max machines are the real protagonists of this story — benefiting from a miracle one person plus AI created in 7 days.
---
References
Postscript: What That include Taught Me
After finishing this article, I went back to Tom's Day 1 log. That include that silently made the Metal compiler fall back to CPU reminded me of countless similar pitfalls I've stepped on.
Engineering isn't about big breakthroughs; it's about small details.
A wrong include, an under-precise intermediate variable, a missed edge case — these are what separate "runs" from "runs well."
Tom proved in 7 days that AI can help you write code, but judging whether code is right, why it's wrong, and how to fix it — that still takes a human.
At least for now.
---
*Written April 5, 2026, based on Tom Turney's turboquant_plus project documentation and community discussions.*