—— Or: why the parameters we've used for two years were all wrong
> *Feynman would say: if you can't explain it simply to someone at a bar, you don't really understand it.*
---
Imagine you have two computers
One is a 4090 with 24GB VRAM — it slices through every model like butter. The other is an ordinary laptop with 8GB VRAM — the kind you'd assume "can't run big models."
Now the question: Qwen3-30B-A3B, a 30-billion-parameter mixture-of-experts model, is generally believed to require at least 16GB of VRAM. But if you know one thing, that 8GB machine can hit 21 tok/s.
Not 3 tok/s. 21.
A 7x gap. The cause isn't the GPU — it's your understanding of "where the model actually runs."
---
A model isn't a brick, it's an apartment building
Many people imagine a large model as a brick that must fit entirely into VRAM. Wrong. Modern models, especially MoE architectures like Qwen3, are more like an apartment building:
- Some residents go out to work every day (attention layers, compute-intensive)
- Some residents lie at home taking up space (MoE expert layers, many parameters but mostly inactive)
- f16: a fine notebook, very legible, but thick
- iso3: compressed shorthand, thin notebook, but decompression costs time when looking back
- 8K context: ubatch 512 beats 128 by 7.6%
- 64K context: ubatch 512 beats 128 by 21.6%
By default, tools like LM Studio stuff the whole building into VRAM — including the residents who lie around all day. Result: 8GB VRAM is 7549MB full (93%), speed is only 3 tok/s, because the GPU is fighting a pile of inactive parameters for space.
What's the smart move?
Keep the workers on the GPU (they need to compute fast) and move the sleepers to CPU memory. Attention layers run on GPU; MoE expert layers go to CPU. VRAM usage drops from 7549MB to 2603MB (32%), and speed jumps from 3 tok/s to 21 tok/s.
7x. 65% less VRAM — and faster.
This sounds counterintuitive — isn't CPU slower than GPU? Yes, but MoE expert layers have a secret: only a small subset of experts activates each time. Keeping inactive experts on CPU and fetching them when needed costs far less than letting the GPU get crushed by idle parameters.
---
KV cache: VRAM's notebook
Imagine reading a thick book where each page requires remembering everything before it. The KV cache is the model's notes on "everything read so far."
Different versions of those notes exist:
The benchmarks are honest:
| Notes format | Speed | |----------|------| | iso3 (3-bit compressed) | 19.4 tok/s | | f16 (half precision) | 51.7 tok/s |
f16 is nearly 3x faster than iso3.
So why not always use f16? Because the notebook is too thick — 8GB VRAM can't fit it. The correct strategy: first compute how many pages f16 needs; use f16 if it fits, degrade otherwise.
The formula is simple: KV VRAM = 2 × layers × KV heads × head dim × context length × bytes per element / 1024²
But there's a trap: many models use GQA (Grouped Query Attention), where KV heads are far fewer than attention heads. If you compute with attention heads, you'll overestimate by 3-4x and waste usable context length.
---
The trap of community formulas
A VRAM estimation formula by oobabooga circulates widely in the community, used to predict "how much context you can run after loading the model."
The problem: it's fitted on q8_0 and f16. With iso3 (3-bit compression), the formula severely overestimates VRAM needs. The result? You could run 64K context, but the formula says 4K.
I tried formula-based prediction and it failed. So I gave up on formulas and switched to binary probing: start from the maximum plausible value, halve on OOM, at most 5 probes, and let llama-server itself tell you what fits.
Sounds dumb? Yes. But far better than a "precise wrong answer."
---
Parallel slots: your multithreading is actually slicing the cake
llama.cpp enables 4 parallel slots by default for multi-user concurrency. But if you're a single user, what are those 4 slots doing?
They're splitting your VRAM into 4 parts. Each gets its own KV cache. For one user, the other 3 just sit there.
After disabling extra slots (--parallel 1): 18.5 → 38.2 tok/s, a direct doubling.
It's like a restaurant that sets 4 plates for you by default. You ordered one dish, but 4 plates occupy the table. Remove 3, and your dish has more room.
---
ubatch: no universal optimum
ubatch is the number of tokens processed per batch during inference. In theory, larger ubatch means more parallelism and more speed. But measured results:
The conclusion is simple: just benchmark both values and pick the faster one. Far more reliable than guessing from documentation.
---
Context compression: don't use the model to compress the model
This is where I stumbled hardest.
What to do when context is full? My initial approach: call the local model to generate a summary. Elegant, right? Let AI compress itself.
Result: single-slot blocking, straight to timeout. Think about it — the model is busy answering your question, and you suddenly ask it to stop and write a summary? It says "let me finish this line of code," and your compression request just hangs there.
It's like writing a long email and having your assistant say halfway through, "Sir, please first write a summary of this email." You: ???
The fix: pure algorithmic extraction. Keep the head (system prompt + first turn) and tail (most recent 8K tokens), keep middle sections by keyword weight — code paths, function names, filenames, TODOs, command lines. 73% compression, <1ms latency.
73% compression, under 1 millisecond. Thousands of times faster than model self-summarization, and non-blocking.
---
So what does this tool do?
It automates all these "manually tuned" parameters.
One command to launch, automatically: 1. Identify model architecture (MoE? GQA? how many layers?) 2. Compute f16 KV cache VRAM footprint 3. Degrade if it doesn't fit; use the fastest option if it does 4. Probe maximum context length (binary search, not formula guessing) 5. Identify MoE expert layers and offload them to CPU automatically 6. Benchmark ubatch 128 vs 512 and pick the winner 7. Compress context algorithmically in <1ms when full
No changes to the inference engine — just parameter decisions. But with the right decisions, an 8GB GPU delivers speeds people assume require 24GB.
---
One-line takeaway
Most people waste their GPU with default parameters. It's not that the hardware isn't enough — it's that you don't know what the parameters should look like.
It's like buying a sports car and leaving it in first gear. Pressing the gas won't help. Take time to understand "a model is an apartment building," "the KV cache is a notebook," "parallel slots slice the cake" — then encode that understanding into an automation tool.
The tool is called Kaiwu, open-sourced on GitHub: https://github.com/val1813/kaiwu
OpenAI-compatible API, plugs directly into Continue / Cursor / Claude Code.
---
*Local inference optimization through a Feynman lens.*