Key points
- The impossible equation made possible: GLM-5.2 has 744B parameters. At int4 quantization (0.5 bytes per parameter), the full model needs 372 GB—15× more than a 25 GB laptop. colibrì (Apache 2.0, ~1,300 lines of pure C, zero dependencies) by developer JustVugg proves this is solvable by exploiting MoE sparsity.
- The loophole: MoE architecture sparsity: GLM-5.2 is a Mixture-of-Experts model with 256 experts per layer (8+1 active per token). Only ~40B parameters participate in any single token's forward pass. 40B × 0.5 bytes = 20 GB, which fits in 25 GB RAM. The 94.6% of inactive parameters can remain on disk.
- Three-tier storage hierarchy:
- RAM (9.9 GB resident): attention/MLA layers, shared experts, embeddings, hot-expert LRU cache
- NVMe SSD (~370 GB): 21,504 routed experts at ~19 MB each (int4), streamed on demand
- VRAM (optional): for GPU acceleration
- Why llama.cpp's mmap falls short on MoE: OS page-cache operates at page granularity (4 KB), not expert granularity (19 MB ≈ 4,750 pages), causing partial-expert eviction. OS LRU is unaware of expert identity or routing hotness. OS readahead is sequential while MoE access is random.
- colibrì's expert-aware caching strategy:
- Per-layer LRU caches evict whole 19 MB expert units
- OS page cache retained as a free L2
- Async expert readahead overlaps I/O with compute
- Router-lookahead prefetch exploits 71.6% routing predictability
- Pinned hot-store for frequently activated experts
- MLA attention with 57× KV-cache compression: Multi-head Latent Attention compresses per-token KV cache from 32,768 floats to 576. KV state is persisted to
.coli_kvfiles, enabling restart-resume with byte-exact state and zero re-prefill. - MTP speculative decoding at 2.2-2.8 tokens/forward: Multi-Token Prediction heads let the main model verify multiple draft tokens per forward pass. Two critical constraints documented as defaults: MTP heads must use int8 (int4 collapses acceptance to 0–4%), and draft and verification must compute the same kernel family (
SPEC_PIN=1). - Honest benchmarks (slow but correct): Forward passes are token-exact, matching the transformers reference implementation 32/32.
- Three layers of disruption:
- *Technical*: redefines "loading a model" from "reading all weights into RAM" to "building an on-demand streaming pipeline."
- *Industry*: MIT-licensed frontier weights run on consumer hardware; code and conversations stay local, eliminating API costs—critical for privacy-sensitive domains.
- *Philosophical*: "Not renting intelligence behind an API—holding it: probing it, measuring it, improving it."
- Three engineering principles extracted: 1. Know what the OS doesn't—domain knowledge beats generic abstractions by orders of magnitude. 2. Measure quality, don't assume it—every performance number is sourced (issue numbers, experiment logs). 3. Defaults encode hard-won lessons—the two MTP defaults exist because every new user would otherwise hit the same pitfalls.
- Deeper observation: MoE decouples parameter count from memory requirement. Parameter count sets the intelligence ceiling; memory requirement sets the deployment cost. colibrì proves these can differ by 15× when storage is fast enough and caching is smart enough.
- Roadmap from "runnable" to "usable": better prefetch (target 90%+ predictability), GPU-accelerated matmul for dense layers (10–50× speedup), parallel expert loading, and structured pruning of never-activated experts.
- Project: https://github.com/JustVugg/colibri (Apache 2.0)
- Model: GLM-5.2 (Zhipu AI, MIT license)
- Engine: pure C, zero dependencies, ~1,300 lines of core code
- Reference video: https://www.youtube.com/watch?v=19xCOJxWU0A
| Hardware | Speed | Notes | |----------|-------|-------| | 6× RTX 5090 (fully resident) | 5.8–6.8 tok/s | TTFT ~13s | | 128 GB CPU desktop | ~1.8 tok/s | warm cache | | M5 Max (128 GB, Metal) | 1.06–1.83 tok/s | | | Ryzen AI 9 HX 370 (128 GB) | 0.37 tok/s | | | 25 GB dev machine | 0.05–0.1 tok/s | cold start, honest baseline |
Cloud H100 inference runs 30–50 tok/s—colibrì is 10–100× slower, but correct.