TurboQuant+ Deep Dive: A Former Google Engineer Takes On Big Tech—Alone

> Style reference: Paul Graham's essay prose + Feynman's physics intuition

Prologue: An Absurd Opening

On March 24, 2026, Google Research published a paper with a bland title: *TurboQuant: Redefining AI Efficiency with Extreme Compression*. But the content landed like a depth charge—they compressed the KV cache during LLM inference down to 3 bits, cut memory usage to one-sixth, achieved 8x speedup, and claimed "zero accuracy loss."

Within 48 hours, memory chip stocks shed billions of dollars in market value. Micron and SK Hynix shares fell off a cliff.

But Google released no code.

"Here's the paper, the principle is explained—have fun implementing it yourselves."

When Tom Turney saw the news, he was sitting in his apartment. He had just left Google after 13 years; his LinkedIn said "independent researcher." Staring at the paper, he had one thought:

"If this thing is real, it should run on every MacBook."

Three days later, he open-sourced turboquant_plus. Six days in, the project had 1,274 stars. On day seven, you could run a 104B-parameter model on an M5 Max with 128K context, using less memory than before.

And he did it "with" Claude Code—as Tom noted on Twitter: "When I say 'I,' it's actually me with Claude Code and Codex—the main work is a lot of steering and babysitting."

This isn't a story about "will AI replace programmers." It's a story about how much one person plus AI can demolish of conventional assumptions.

---

Part 1: The KV Cache—LLM Inference's Memory Burden

To understand why TurboQuant matters, you must first understand the memory black hole of LLM inference.

Imagine reading a thick book where you must remember every previous page to understand context. Transformers work this way—attention lets the model "look back" at all previous tokens. To avoid recomputing, it stores the computed "Key" and "Value" vectors: the KV cache.

The problem: this cache grows linearly with context length.

1,000 tokens? Small cache. 100,000 tokens? Tens of GB. Flagship models now support million-token contexts, and the KV cache is the biggest bottleneck—not compute, but memory capacity and bandwidth.

The traditional fix is quantization: compressing 16-bit floats to 8-bit, 4-bit, or lower. But quantization has costs—compress too hard and the model starts babbling.

Google's TurboQuant claimed to break this trade-off: 3 bits, 6x compression, zero loss.

---

Part 2: PolarQuant's Intuition—Why Direction Matters More Than Magnitude

TurboQuant's core is an algorithm called PolarQuant. Tom saw a beautiful mathematical intuition in the paper:

**In attention computation, vector *direction* matters far more than vector *magnitude*.

Think of navigation. Your target is 5 km east—"east" is direction, "5 km" is distance. If I change it to 4.8 km, you'll still find the place. But if I'm off by 30 degrees, you're completely lost.

Attention is fundamentally dot products, and dot product = direction similarity × magnitude product.

PolarQuant converts Cartesian coordinates (x, y) into polar coordinates (angle θ, radius r). Then it compresses the radius brutally (it doesn't matter much) and carefully protects the angle.

Even better: a Walsh-Hadamard transform—a rotation that "shuffles" vector dimensions so unevenly distributed values become Gaussian-like and uniform. Uniformity means smaller quantization error.

When Tom read this, his eyes lit up. Not just an engineering trick—a victory of geometric intuition.

---

Part 3: Day 1 Marathon—90 Commits and the Curse of One include

Tom decided to implement the algorithm from scratch—not a quick PyTorch demo, but a full integration into llama.cpp, the open-source inference engine that runs LLMs on laptops.

Day 1 was a marathon: 90 commits, morning to midnight.

By evening, the first Metal GPU kernel ran. Tom pressed Enter with high hopes: 2.4 tokens/second.

An M5 Max should hit 85.5 tok/s. 2.4? Slower than CPU.

After hours of debugging, the culprit was a single include line that made the Metal compiler silently fall back to CPU mode—no error, the program "looked" like it was running on GPU while softly spinning its wheels.

Fixed, speed jumped to 51.4 tok/s. But before he could celebrate: PPL = 165.6.

Perplexity measures output quality; a normal value is 6–8. 165.6 means pure gibberish. Tom later joked in his docs: "Speed benchmarks measure how fast the model talks nonsense."

That was Day 1: speed, no quality. But he'd cleared the deepest pit—he knew the problem wasn't the algorithm, just implementation details.

---

Part 4: 36 Hours to Working—From Gibberish to Poetry

Days 2–3 focused on fixing the nonsense.

PolarQuant's rotation step demands high numerical precision. float64 in Python works fine, but on GPU with fp16, accumulated error makes the model collapse. Tom spent long hours tuning intermediate precision, walking a tightrope between speed and correctness.

After 36 hours, the first end-to-end test passed:

• Compression: 4.6x (turbo3)
• PPL: 6.176 vs. q8_0 baseline of 6.111—only 1.06% higher
• Speed: close to q8_0 prefill speeds

The model wrote poems, code, and answers—nearly indistinguishable from full precision.

---

Part 5: Beyond the Paper—Sparse V and Three Unexpected Discoveries

Tom didn't stop at "reproducing the paper." Integrating into llama.cpp, he found three things the original paper never mentioned:

Discovery 1: V compression is "free"

TurboQuant compresses Key and Value caches symmetrically. But Tom found: Value vectors can be compressed far harder with almost no quality impact.

His explanation: in attention, Keys decide "where to look" (routing); Values decide "what information to fetch." Wrong routing ruins everything downstream; slightly blurry Values don't.

Hence asymmetric K/V compression: Keys at q8_0 (8-bit), Values at turbo3 (3-bit). This rescues quantization-sensitive models while capturing most of the memory savings.

Discovery 2: All quality loss comes from K compression

Further experiments confirmed the intuition. Low-precision Keys scramble attention routing and the model "loses focus." Values alone, even at 2-bit, barely change quality.

This explains why some models thrive with TurboQuant and others collapse—it depends on each model's sensitivity to Key precision.

Discovery 3: Boundary layers are especially fragile

The Transformer's first and last few layers are most quantization-sensitive. Middle layers compress freely; boundary layers need gentle treatment.

His Boundary V scheme: q8_0 Value cache for the first 2 and last 2 layers, turbo2 in between. Fifteen lines of code, zero speed cost, recovering 37–91% of quality loss.

---

Part 6: Sparse V Dequant—A 22.8% Speedup Gift

Tom's biggest original contribution is Sparse V Dequant.

At long contexts (e.g., 32K tokens), attention distributions over each new token are extremely sparse—most positions have near-zero attention weights (< 1e-6).

If their contribution is negligible, why spend time dequantizing them?

Sparse V uses attention weights as a gate during decoding, skipping dequantization of low-weight Value vectors.

The results:

• 22.8% decode speedup at 32K context
• Zero perplexity change (validated over 50 chunks, CI ±0.021)
• Works not just for TurboQuant, but also q8_0 and q4_0

This is a format-agnostic optimization. Tom: "This isn't a TurboQuant trick—it's an inherent property of attention."

Even more surprising, Needle-in-a-Haystack results: with Sparse V enabled, single-needle retrieval rose from 7/9 to 9/9 (100%). Multi-key retrieval hit 100% at 32K context.

Tom speculates that skipping low-weight quantization noise acts as denoising.

---

Part 7: The 7-Day Timeline—A Race Against the Clock

| Time | Milestone | |------|-----------| | Day 1 | Python prototype + llama.cpp integration, 90 commits, Metal shader pitfalls, PPL 165.6 nightmare | | Days 2–3 | Precision fixes, 141-line core stabilized, 500+ unit tests, 100% coverage | | Days 3–5 | C port + Metal GPU kernels, end-to-end run, speed optimization begins | | Days 5–7 | Extreme optimization: fp16 half-precision, half4 vectorized butterflies, graph-side rotations, block-32 storage layout |

Final results:

• 511+ Python tests, 100% coverage
• C port integrated into llama.cpp
• Metal GPU kernels for Apple Silicon M1–M5
• Community validation: CUDA (RTX 3080/3090/4090/5090), AMD (RX 9070 XT)
• Speed: 739 tok/s → 2,747 tok/s, a 3.7x improvement, on par with q8_0

---

Part 8: The Full Performance Picture

Compression vs. Quality

| Config | Bits/value | Compression | PPL (wikitext-2) | vs. q8_0 | |--------|-----------|-------------|------------------|----------| | f16 | 16.0 | 1.0x | 6.121 | -0.16% | | q8_0 | 8.5 | 1.9x | 6.111 | baseline | | turbo4 | 4.25 | 3.8x | 6.125 | +0.23% | | turbo3 | 3.5 | 4.6x | 6.176 | +1.06% | | turbo2 | 2.5 | 6.4x | 6.507 | +6.48% |

turbo4's quality is essentially identical to q8_0 (the difference is within noise). A victory for PolarQuant's geometric intuition over brute-force quantization.

Large Model Stress Tests (M5 Max 128GB)

| Model | Params | Weights | KV config | PPL | Max context | NIAH | |-------|--------|---------|-----------|-----|-------------|------| | Llama-3.1-70B | 70B | Q4_K_M | turbo4/turbo4 | 3.461 | 48K | 30/30 | | Command-R+ | 104B | Q4_K_M | turbo3/turbo3 | 6.415 | 128K | 10/10 |

A 104B model with 128K context, on a MacBook. A month ago, fantasy.

Sparse V Magic

Qwen3.5-35B-A3B (MoE) decode speed at 32K context:

| Config | Short text | 32K context | vs. q8_0 | |--------|-----------|-------------|----------| | q8_0 | 85.71 tok/s | 1173.91 tok/s | baseline | | turbo3 | 76.84 tok/s | 1141.74 tok/s | 0.90x | | turbo3 + Sparse V | ~76 tok/s | ~1400 tok/s | ~1.19x |

With Sparse V, long-context decoding overtakes q8_0.

---

Part 9: What It Means—One Person + AI = ?

The most striking thing about Tom Turney's project isn't the technical detail—it's the new engineering paradigm it demonstrates.

Lesson 1: AI is a lever, not a replacement

Tom didn't let Claude Code "write his code." He:

• Read the paper and understood the core mathematical intuition
• Designed the architecture: Python for validation, C for production
• Diagnosed problems (that damned include)
• Made judgment calls: precision problem or algorithm problem?

Claude Code was his executor and sparring partner. Human intuition + AI generation = output exceeding either alone.

Lesson 2: Big labs publish papers; small teams ship

Google has thousands of engineers, but TurboQuant's code—if they ever release it—might take months of reviews, compliance checks, and internal politics.

Tom: one person, 7 days.

Not because Google's engineers can't. Because small teams iterate faster than large organizations' bureaucratic inertia. When AI-assisted coding compresses implementation from weeks to days, the advantage of moving fast is magnified tenfold.

Lesson 3: The power of open source

turboquant_plus isn't Tom's work alone. From Day 3, the community poured in:

• CUDA validation on RTX 4090 and RTX 3090
• Independent verification of Boundary V on NanoGPT
• AMD RX 9070 XT support contributed by community members

6 days, 174 commits, 1,274 stars, 30+ testers across M1/M2/M3/M5 Macs, NVIDIA, and AMD.

That's the terrifying power of open source—once the seed is planted, it grows exponentially.

Lesson 4: "Beyond the paper" becomes the norm

Tom didn't just reproduce the paper—he shipped Sparse V, Asymmetric K/V, Boundary V, and Temporal Decay, none of which were in the original work.

This will become more common. As implementation cost falls, every engineer can be a researcher—experimenting while implementing, discovering while benchmarking.

Papers are no longer endpoints. They're starting points.

---

Epilogue: What We're Really Talking About

We're not talking about a KV cache compression algorithm.

We're talking about one person deciding to take on Big Tech—and winning.

We're talking about AI as an engineering lever that lets individuals break past organizational boundaries.

We're talking about open source turning a paper into a production tool that runs 104B models on laptops—in 7 days.

From Tom's README:

> "If individual modules prove useful and stable, the goal is to gradually upstream them into llama.cpp as small, reviewable patches."

A humble ambition. Not a separate kingdom—advancing the whole ecosystem.

The 1,274 stars are the community's answer. And the developers running 128K contexts on M5 Max hardware are the real protagonists of this story.

---

References

• turboquant_plus: https://github.com/TheTom/turboquant_plus
• llama.cpp fork: https://github.com/TheTom/llama-cpp-turboquant
• TurboQuant paper: arXiv:2504.19874 (ICLR 2026)
• PolarQuant paper: arXiv:2502.02617 (AISTATS 2026)
• Google Research Blog: TurboQuant: Redefining AI Efficiency

Engineering isn't about big breakthroughs—it's about small details.

A wrong include, an under-precise intermediate variable, an overlooked edge case—these are what separate "runs" from "runs well."

In 7 days, Tom proved: AI can write your code, but judging whether it's right, why it's wrong, and how to fix it**—that still needs a human.

For now, at least.

*Written April 5, 2026, based on the turboquant_plus project documentation and community discussions.*