colibri: How 1,300 Lines of C Run a 744B MoE LLM on a 25 GB Laptop

Key points

• The impossible equation. A naive int4 footprint of a 744B-parameter model is 744B × 0.5 bytes ≈ 372 GB, far exceeding a 25 GB laptop. colibri sidesteps this through MoE sparsity, not compression tricks.
• MoE as a memory loophole, not just a parameter-efficiency trick. GLM-5.2 routes 8 of 256 experts per layer per token. The active parameter set per token is roughly 40B, or 5.4% of total weights, which at int4 is about 20 GB — small enough to fit. The other 94.6% of weights are cold data that can live on NVMe SSD.
• Three-tier storage pyramid.
• VRAM (optional, fastest)
• RAM (~9.9 GB resident): MLA attention layers, shared experts, embeddings, hot-expert LRU
• NVMe SSD (~370 GB): all 21,504 routed experts, ~19 MB each in int4, streamed on demand
Per-layer LRU evicts whole experts rather than OS pages, so a 19 MB block never gets half-evicted.
• Why llama.cpp mmap falls short for MoE. OS page cache evicts by 4 KB pages with no notion of expert boundaries, does not know which pages belong to frequently called experts, and cannot do random-access async readahead. colibri replaces OS-level decisions with expert-aware caching plus async readahead and a router-lookahead prefetch that exploits the 71.6% predictability of routing decisions.
• MLA attention: 57x KV-cache compression. KV cache per token drops from 32,768 floats to 576. colibri persists the KV cache to .coli_kv files, so a session can be resumed with byte-level consistency and zero re-prefill.
• MTP speculative decoding. colibri runs GLM-5.2's native Multi-Token Prediction head, achieving 2.2-2.8 accepted tokens per main-model forward, roughly doubling effective throughput on CPU. Two hard-won defaults:
• MTP head must be quantized to int8 (int4 collapses acceptance to 0–4%, per issue #8)
• Draft and verifier must share the same kernel family (SPEC_PIN=1); mismatched numerics silently degrade acceptance (forensic write-up in issue #163).
• Honest benchmarks. Token output is verified token-exact against a transformers reference implementation (32/32 match). Speed varies by hardware:
• 6× RTX 5090 (fully resident): 5.8–6.8 tok/s, TTFT ~13 s
• 128 GB CPU desktop, warm cache: ~1.8 tok/s
• M5 Max (128 GB, Metal): 1.06–1.83 tok/s
• Ryzen AI 9 HX 370 (128 GB): 0.37 tok/s
• 25 GB dev machine, cold start: 0.05–0.1 tok/s
Cloud H100 inference is 30–50 tok/s; colibri is 10–100x slower but correct.
• Reframing "loading a model." Traditional engines equate loading with fitting all weights in RAM. colibri equates it with building an on-demand streaming pipeline whose working set equals the per-token active footprint, not the total parameter count. MoE therefore decouples parameter count (capability ceiling) from memory requirement (deployment cost) — 15x apart in this case.
• Three engineering principles surfaced by the project.
1. Bypass generic abstractions when you have domain knowledge — expert-aware caching beats OS page caching. 2. Measure quality, do not assume it; every number in the README cites an issue or log. 3. Good defaults encode the bugs other users would otherwise hit; the int8 MTP and same-kernel-pin rules were earned in debugging.
• Implications. Open-weights frontier models are no longer locked to datacenters. GLM-5.2 is MIT-licensed by Zhipu AI; combined with colibri, anyone can run a 744B model locally, keep data on-device, and avoid API costs. This is significant for privacy-sensitive domains (medical, legal, defense).
• Roadmap. Better router-lookahead prefetch (push predictability above 90%), GPU-accelerated matmul for dense layers (10–100x potential), parallel expert loading, and structured pruning of never-activated experts to cut the 370 GB on-disk footprint further.

References

• Project: JustVugg/colibri (Apache 2.0)
• Model: GLM-5.2 (Zhipu AI, MIT license)
• Engine: pure C, zero dependencies, ~1,300 lines of core code
• Source video: Devsplainers on YouTube
• MLA technique origin: DeepSeek-V3