How colibrì Runs a 744B-Parameter LLM in 25 GB RAM: 1,300 Lines of C Code

Forum topic · ✨步子哥 · 2026-08-03

Summary

colibrì, a 1,300-line dependency-free C inference engine by JustVugg, runs the 744-billion-parameter GLM-5.2 MoE model on a 25 GB laptop with no GPU. The trick exploits MoE sparsity: only about 5.4% of parameters (roughly 40B) fire per token, so int4 quantization fits the active weights in 20 GB. colibrì streams the 21,504 routed experts (~19 MB each, ~370 GB total) from NVMe SSD on demand, replacing llama.cpp's mmap strategy with a per-layer LRU that understands expert boundaries and uses async readahead plus router-lookahead prefetch (71.6% predictable). MLA attention compresses KV cache 57x and persists it to disk for byte-exact resume. MTP speculative decoding achieves 2.2–2.8 tokens per forward, but the MTP head must stay at int8 or acceptance collapses to 0–4%. Benchmarks range from 5.8–6.8 tok/s on 6× RTX 5090 down to 0.05–0.1 tok/s on 25 GB cold cache, all token-exact versus the transformers reference.

Key points

The impossibility equation, reworked. GLM-5.2 has 744B parameters; at int4 (0.5 byte each) the full model is 372 GB, which does not fit in 25 GB of RAM. colibrì reframes the problem: with MoE sparsity, only ~40B parameters activate per token (~5.4%), so 40B × 0.5 B = 20 GB fits. The goal is no longer "load the whole model," but "build an on-demand pipeline for the slice that matters."
MoE sparsity is the loophole, not the answer. Because the router chooses 8 of 256 experts per token dynamically, any of those experts may live on NVMe. colibrì's core engineering challenge is keeping SSD reads fast enough that expert fetches don't dominate latency.
Three-tier storage hierarchy.
VRAM (GPU, optional, fastest): fastest path when present.
RAM (~9.9 GB resident): attention (MLA), shared experts, embeddings, plus an LRU of hot experts.
NVMe SSD (~370 GB): all 21,504 routed experts (~19 MB each at int4), streamed on demand into per-layer LRU caches. Only about 11 GB of parameters change between tokens — the routed experts; the other ~34 GB is dense and stays in RAM.
Why llama.cpp mmap falls short on MoE. OS page cache is page-granular (4 KB) and oblivious to expert boundaries (~4,750 pages per expert), so it can evict half an expert. It also lacks expert-aware popularity signals and does no async readahead for random access patterns. colibrì replaces this with expert-aware caching while keeping the OS page cache as a free L2.
colibrì's cache toolkit. Per-layer LRU that evicts whole 19 MB expert blocks (no "half expert" problem), async expert readahead during the current forward pass, experimental router-lookahead prefetch exploiting 71.6% routing predictability, and a pinned hot-store for the most-used experts.
MLA attention and persistent KV cache. Multi-head Latent Attention compresses each token's KV from 32,768 floats to 576 — a 57× compression. colibrì writes KV state to .coli_kv files, enabling byte-exact resume across restarts (zero re-prefill), which is critical because prefill is the slowest phase on a 25 GB box.
MTP speculative decoding, with two hard rules. The Multi-Token Prediction head is used as a drafter; the main model verifies batches of draft tokens. colibrì averages 2.2–2.8 accepted tokens per forward, lifting a 0.3 tok/s system to ~0.7–0.8 tok/s effective. Two non-negotiable defaults are documented in the README: the MTP head must be int8 (int4 collapses acceptance to 0–4%, see issue #8), and drafter and verifier must compute the same function (pinned via SPEC_PIN=1; see issue #163).
Honest benchmarks. Speed ranges from 5.8–6.8 tok/s on 6× RTX 5090 (TTFT ~13 s), to ~1.8 tok/s on a 128 GB CPU desktop (warm cache), 1.06–1.83 tok/s on M5 Max (Metal), 0.37 tok/s on Ryzen AI 9 HX 370, and 0.05–0.1 tok/s cold on a 25 GB dev box. Cloud H100 inference is 30–50 tok/s, so colibrì is 10–100× slower — but token-exact against the transformers reference (32/32), not approximate.
Three engineering principles.

Know what the OS doesn't.

Measure quality, don't assume it.

docs/benchmarks.md

Defaults are scars.

MoE changes the meaning of "loading a model." Dense models use 100% of parameters per token, so loading = reading everything. MoE uses ~5.4%, so ~94.6% is cold data at any instant — and cold data does not need to live in RAM. This decouples parameter count (intelligence ceiling) from memory footprint (deployment cost); colibrì demonstrates a 15× gap.
Implications.
*Technical:* on-demand expert streaming is viable and redefines "load" for MoE.
*Industry:* frontier open-weight models are no longer gated by data-center hardware; MIT-licensed GLM-5.2 + colibrì runs locally, privately, with zero API bill — relevant for medical, legal, and defense workloads.
*Philosophy:* the README's framing — "not renting intelligence behind an API, but holding it: probing it, measuring it, improving it" — captures the real stake of open weights.
Roadmap toward "usable." Better prefetch (push router-lookahead beyond 71.6%), GPU-accelerated matmul for dense layers, parallel multi-expert I/O, and structured pruning to drop never-activated experts from disk.

Resources

Project: https://github.com/JustVugg/colibri (Apache 2.0)
Model: GLM-5.2 (Zhipu AI, MIT license)
Engine: pure C, zero dependencies, ~1,300 lines of core code
Source video: https://www.youtube.com/watch?v=19xCOJxWU0A

How colibrì Runs a 744B-Parameter LLM in 25 GB RAM: 1,300 Lines of C Code

Summary

Key points

Resources

Tags