colibrì: Running a 744B-Parameter MoE Model in 25GB RAM with ~1,300 Lines of C Code

Forum topic · ✨步子哥 · 2026-07-26

Summary

colibrì is a dependency-free inference engine in roughly 1,300 lines of C that runs the 744-billion-parameter GLM-5.2 MoE model on a laptop with only 25GB of RAM and no GPU. It exploits MoE sparsity: only about 40B parameters (5.4%) are activated per token, so 40B × 0.5 bytes (int4) fits in RAM while 21,504 routed experts (~370GB, ~19MB each) are streamed on demand from NVMe SSD. colibrì bypasses the OS page cache with expert-level caching: per-layer LRU eviction of whole experts, async readahead, router-lookahead prefetch (71.6% routing predictability), and pinned hot experts. It implements MLA attention (57× KV-cache compression) with disk-persisted KV state for byte-exact session resume, and MTP speculative decoding achieving 2.2-2.8 tokens per forward pass — requiring an int8 MTP head, since int4 collapses acceptance to 0-4%. Speed ranges from 5.8-6.8 tok/s on 6× RTX 5090 down to 0.05-0.1 tok/s on the 25GB dev machine, but forward passes are token-exact verified (32/32 match vs. the transformers reference). The project argues MoE decouples total parameter count from memory requirements, proving frontier open-weight models can run on ordinary consumer hardware.

colibrì (by GitHub user JustVugg) is a minimal LLM inference engine — ~1,300 lines of C, zero dependencies — that runs GLM-5.2, a 744-billion-parameter model, on a laptop with 25GB of RAM and no GPU. It hit 453 points on Hacker News within three hours. This is a translation and structured summary of the original Chinese analysis.

The math problem

GLM-5.2 has 744B parameters. At int4 quantization (0.5 bytes each), that's ~372GB — 15× more than the available 25GB of RAM. Conventional wisdom said it was impossible without an H100 cluster or cloud APIs.

The loophole: MoE sparsity

GLM-5.2 is a Mixture-of-Experts model with 256 expert subnetworks; per token, only 8+1 experts activate, so only ~40B parameters (~20GB at int4) participate in each forward pass — about 5.4% of the total. The catch: you don't know in advance which experts the router will pick, so the engineering challenge is making disk reads fast enough not to stall inference.

Key points

Three-tier storage hierarchy: ~9.9GB resident in RAM (attention/MLA layers, shared experts, embeddings, hot-expert LRU cache) plus 21,504 routed experts (~19MB each, ~370GB total) streamed from NVMe SSD on demand.
Why llama.cpp's mmap falls short for MoE: access patterns are random rather than sequential. The OS page cache (1) doesn't know expert boundaries (one 19MB expert spans ~4,750 4KB pages, so partial eviction causes rereads), (2) treats all pages equally despite differing expert popularity, and (3) can't do async readahead for unpredictable random access.
colibrì's answer — application-level expert caching: per-layer LRU evicting whole experts, the OS page cache as a free L2, async expert readahead, experimental router-lookahead prefetch (71.6% of routing decisions are predictable), and pinned hot experts that never evict.
MLA attention: Multi-head Latent Attention compresses per-token KV cache from 32,768 floats to 576 — a 57× reduction. KV cache is persisted to disk (.coli_kv), so sessions resume byte-exactly with zero re-prefill after restart.
MTP speculative decoding: the native Multi-Token Prediction head drafts multiple tokens that the main model verifies in one forward pass, achieving 2.2–2.8 tokens/forward. Two hard-won rules: the MTP head must be int8 (int4 collapses acceptance to 0–4%, see issue #8), and draft and verification must use the same kernel family (SPEC_PIN=1, issue #163).

Benchmarks (honest numbers)

| Hardware | Speed | Notes | |---|---|---| | 6× RTX 5090 (fully resident) | 5.8–6.8 tok/s | TTFT ~13s | | 128GB CPU desktop | ~1.8 tok/s | warm cache | | M5 Max (128GB, Metal) | 1.06–1.83 tok/s | | | Ryzen AI 9 HX 370 (128GB) | 0.37 tok/s | | | 25GB dev machine | 0.05–0.1 tok/s | cold start, honest baseline |

Cloud H100 inference runs 30–50 tok/s, so colibrì is 10–100× slower — but correctness is verified: forward passes match the transformers reference implementation 32/32. As the README puts it: "This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan."

Why it matters

Technically, colibrì redefines "loading a model": not reading all weights into memory, but building an on-demand streaming pipeline. The requirement drops from memory ≥ model size to memory ≥ per-token active parameters.
Industrially, MIT-licensed GLM-5.2 weights (370GB) can run fully locally — no API bills, no data leaving the machine — which matters for privacy-sensitive domains.
Philosophically, per the README: "Not renting intelligence behind an API — holding it." A web dashboard visualizes all experts as a live "cortex" colored by storage tier and brightness by routing heat.

Engineering principles highlighted

1. Know what the OS doesn't — domain knowledge (expert boundaries, expert popularity) beats generic page-cache abstractions. 2. Measure quality, don't assume it — every number in the README cites its source (issue numbers, experiment logs). 3. Good defaults are scar tissue — the int8-MTP and same-kernel rules exist because every new user would otherwise hit those traps.

The deeper insight and roadmap

MoE decouples total parameter count from memory requirements: 94.6% of parameters are "cold data" at any moment and can live on SSD. Deployment economics shift from 750GB+ of VRAM to 25GB RAM + 370GB NVMe. colibrì (v1.1.0, Apache 2.0) is still a proof of concept; the roadmap targets better prefetching beyond 71.6% routing predictability, GPU-accelerated matmul (10–50× dense-layer speedup), multithreaded expert loading, and expert pruning.

Links:

Project: https://github.com/JustVugg/colibri (Apache 2.0)
Model: GLM-5.2 (Zhipu AI, MIT license)
Video: https://www.youtube.com/watch?v=19xCOJxWU0A