Why Your 8GB GPU Can Hit 21 tok/s: The Real Story Behind Local LLM Inference Optimization

Forum topic · 小凯 · 2026-04-27

Summary

A detailed write-up on local LLM inference optimization shows how an 8GB GPU can run Qwen3-30B-A3B, a 30B-parameter MoE model, at 21 tok/s instead of 3 tok/s. Key insights: offloading inactive MoE expert layers to CPU while keeping attention layers on GPU cuts VRAM use from 7549MB to 2603MB and yields a 7x speedup; f16 KV cache is nearly 3x faster than 3-bit iso3 compression, so use f16 whenever it fits (accounting for GQA's reduced KV heads); community VRAM estimation formulas like oobabooga's overestimate for low-bit caches, so binary-search probing against OOM is more reliable; setting --parallel 1 doubles throughput (18.5 to 38.2 tok/s) for single users; ubatch gains depend on context length; and algorithmic context compression (73% reduction, under 1ms) beats using the model to summarize its own context. These ideas are implemented in the open-source Kaiwu tool, which automates parameter decisions and exposes an OpenAI-compatible API for editors like Cursor and Continue.

——or, why we've been using the wrong parameters for two years.

> *Feynman would say: if you can't explain it simply to someone at a bar, you don't really understand it.*

---

Imagine Two Computers

One is an RTX 4090 with 24GB VRAM — everything runs like butter. The other is an ordinary laptop with 8GB VRAM — the kind of machine you assume "can't run large models."

Now here's the question: Qwen3-30B-A3B, a 30-billion-parameter mixture-of-experts (MoE) model, is generally believed to require at least 16GB of VRAM. But if you know one thing, that 8GB machine can hit 21 tok/s.

Not 3 tok/s. Twenty-one.

A 7x gap. The cause isn't the GPU — it's your understanding of "where the model actually runs."

---

A Model Isn't a Brick, It's an Apartment Building

Many people imagine a large model as a brick that must fit entirely into VRAM. Wrong. Modern models, especially MoE architectures like Qwen3, are more like an apartment building:

Some residents go out to work every day (Attention layers — compute-intensive)
Some residents just lie around taking up space (MoE expert layers — many parameters, mostly idle)

By default, tools like LM Studio cram the whole building into VRAM — including the residents who never leave. Result: on 8GB VRAM, 7549MB gets occupied (93%), and you get only 3 tok/s, because the GPU is fighting a pile of inactive parameters for space.

What's the smart move?

Keep the working residents on the GPU (they need to compute fast), and move the couch potatoes to CPU memory. Attention layers run on GPU; MoE expert layers go to CPU. VRAM usage drops from 7549MB to 2603MB (32%), and speed jumps from 3 tok/s to 21 tok/s.

7x. 65% less VRAM, and faster.

This sounds counterintuitive — isn't CPU slower than GPU? Yes, but MoE expert layers have a secret: only a small subset of experts activates per token. Keeping inactive experts on CPU and fetching them on demand costs far less than letting the GPU choke on idle parameters.

---

KV Cache: The Model's Notebook

Imagine reading a thick book where each page requires remembering everything before it. The KV cache is the model's notes on "everything so far."

Different versions of those notes exist:

f16: a precise notebook — clear writing, but thick
iso3: compressed shorthand — thin notebook, but decompression takes time when you look back

The benchmark data is honest:

| Cache type | Speed | |------------|-------| | iso3 (3-bit compressed) | 19.4 tok/s | | f16 (half precision) | 51.7 tok/s |

f16 is nearly 3x faster than iso3.

So why not always use f16? Because the notebook is too thick — 8GB VRAM can't always fit it. The correct strategy: compute how much VRAM f16 KV cache requires; use it if it fits, otherwise downgrade.

The formula is simple: KV VRAM = 2 × layers × KV heads × head dim × context length × bytes per element / 1024²

But there's a trap: many models use GQA (Grouped Query Attention), where KV heads are far fewer than attention heads. If you compute with attention heads, you overestimate by 3–4x and waste usable context length.

---

The Community Formula Trap

A VRAM estimation formula by oobabooga circulates widely in the community, used to predict "how much context fits after loading the model."

The problem: it was fitted on q8_0 and f16. With iso3 (3-bit compression), it severely overestimates VRAM needs. The result: you could run 64K context, but the formula tells you only 4K fits.

I tried formula-based prediction — it failed. In the end I abandoned formulas for binary probing: start from the maximum plausible value, halve on OOM, at most 5 probes, and let llama-server tell you what actually fits.

Sounds dumb? Yes. But it beats a precisely wrong answer.

---

Parallel Slots: What You Think Is Multithreading Is Actually Slicing the Cake

llama.cpp defaults to 4 parallel slots for multi-user concurrency. But if you're a single user, what are those 4 slots doing?

They're slicing your VRAM into 4 pieces, each with its own KV cache. With one user, 3 pieces sit idle.

After turning off extra slots (--parallel 1): 18.5 → 38.2 tok/s. Doubled.

It's like a restaurant that sets 4 place settings by default. You ordered one dish, but 4 plates occupy the table. Remove 3, and your dish has more room.

---

ubatch: No Universal Optimum

ubatch is the batch size of tokens processed per inference step. In theory, larger ubatch means more parallelism and more speed. But real measurements:

8K context: ubatch 512 is 7.6% faster than 128
64K context: ubatch 512 is 21.6% faster than 128

Why the difference? At small contexts, the bottleneck isn't compute. At large contexts, KV cache memory access becomes the bottleneck, and larger ubatch merges multiple queries' memory accesses into contiguous blocks, improving cache hit rate.

Conclusion: just benchmark both values and pick the faster one. Far more reliable than guessing from docs.

---

Context Compression: Don't Use the Model to Compress the Model

This is where I stumbled hardest.

Context full — what do you do? My first plan: call the local model to generate a summary. Elegant, right? Let the AI compress itself.

Result: the single slot blocks, and the request times out. The model is busy answering your question, and you suddenly ask it to stop and write a summary? It says "let me finish this line of code," and your compression request just hangs.

It's like writing a long email and having your assistant interrupt: "Sir, please first write a summary of this email." You: ???

The fix: pure algorithmic extraction. Keep the head (system prompt + first turn) and tail (most recent 8K tokens); in the middle, keep by keyword weight — code paths, function names, filenames, TODOs, command lines. Compression ratio: 73%, time: <1ms.

73% compression in under 1 millisecond — thousands of times faster than model self-summarization, and it never blocks.

---

So What Does the Tool Do?

It automates all of these "manually tuned" parameters.

One command to launch, automatically: 1. Identify model architecture (MoE? GQA? how many layers?) 2. Compute f16 KV cache VRAM requirements 3. Downgrade if it doesn't fit; use the fastest if it does 4. Probe maximum context length (binary search, not formula guessing) 5. Identify MoE expert layers and auto-offload them to CPU 6. Benchmark ubatch 128 vs 512 7. Compress full context with the <1ms algorithmic method

No changes to the inference engine — just parameter decisions. But with the right decisions, an 8GB GPU delivers the speed people assume requires 24GB.

---

One-Sentence Summary

Most people waste their GPU on default parameters. It's not that the hardware isn't enough — it's that you don't know what the parameters should be.

It's like buying a sports car and always driving in first gear. Pressing the gas won't help. Spend some time understanding "a model is an apartment building," "KV cache is a notebook," "parallel slots slice the cake" — then encode that understanding into automation.

The tool is called Kaiwu, open source on GitHub: https://github.com/val1813/kaiwu

OpenAI-compatible API — plugs directly into Continue / Cursor / Claude Code.

---

*Buzige | Local inference optimization from a Feynman-style perspective*