colibrì: Running a 744B-Parameter GLM Model in 25GB RAM with 1,300 Lines of C Code

Forum topic · ✨步子哥 · 2026-07-26

Summary

colibrì, an open-source inference engine written in about 1,300 lines of dependency-free C, runs the 744-billion-parameter GLM-5.2 MoE model on a laptop with only 25GB of RAM and no GPU. The trick exploits Mixture-of-Experts sparsity: while the int4-quantized model totals roughly 372GB, only about 5.4% of parameters (~40B) activate per token, which fits in memory. colibrì streams 21,504 routed experts (~19MB each) from NVMe SSD using per-layer LRU caching, async readahead, and router-lookahead prefetching—bypassing the OS page cache, which cannot reason about expert boundaries. It also implements MLA attention (57x KV-cache compression with disk persistence) and int8 MTP speculative decoding achieving 2.2-2.8 tokens per forward pass. Speeds range from 5.8 tok/s on six RTX 5090s to 0.05 tok/s on a 25GB machine, with token-exact verification against reference implementations. The project demonstrates that MoE decouples total parameter count from memory requirements, making frontier-class open-weight models runnable on consumer hardware.

An Impossible Equation

GLM-5.2 has 744 billion parameters. With int4 quantization (0.5 bytes per parameter), that's 372 GB. A typical laptop has 25 GB of RAM. Before July 10, 2026, every LLM inference engine answered this equation with: impossible. Your options were an H100 cluster or a cloud API bill.

Then a developer called JustVugg released colibrì—about 1,300 lines of C code, zero dependencies—running GLM-5.2 on a 25 GB RAM, GPU-less laptop. It hit 453 points on Hacker News within three hours.

This isn't magic. It's exploiting a loophole.

The Loophole: MoE Sparsity

GLM-5.2 is not a dense model. It uses Mixture-of-Experts (MoE): 256 expert subnetworks per layer, with 8+1 activated per token. For any given token, only ~40 billion parameters do the work while 704 billion sleep.

40 billion × 0.5 bytes = 20 GB. 20 GB < 25 GB. Mathematically feasible.

The loophole: you don't need the whole model in memory—only the 5.4% each token actually uses.

But there's an engineering problem: the router dynamically decides which 8 experts to call for each token. The experts you need may live on SSD, and must be loaded within milliseconds.

Three-Tier Storage Hierarchy

colibrì treats storage as a pyramid:

VRAM (GPU): optional, fastest
RAM (~9.9 GB resident): attention layers (MLA), shared experts, embeddings, hot-expert LRU cache
NVMe SSD (~370 GB): 21,504 routed experts, ~19 MB each (int4), streamed on demand with per-layer LRU caching

Key insight: only ~11 GB of parameters change between tokens—the routed experts. The other ~34 GB are dense layers that stay in memory permanently.

Why llama.cpp's mmap Falls Short

llama.cpp uses mmap for weight loading—perfect for dense models with sequential access, but broken for MoE's random access patterns:

1. The OS page cache doesn't know expert boundaries. A 19 MB expert spans ~4,750 4 KB pages; OS LRU can evict half an expert. 2. OS LRU ignores expert popularity. colibrì's experiments show 71.6% of routing is predictable, but the OS can't make expert-level caching decisions. 3. No async prefetch. mmap readahead is sequential; the OS can't know which experts the next token needs.

colibrì's solutions:

Per-layer LRU caches evicting whole experts (19 MB blocks)
OS page cache as a free L2
Async expert readahead during current-token compute
Router-lookahead prefetch (experimental, exploiting 71.6% routing predictability)
Pinned hot-store for frequently used experts

MLA Attention: 57x KV-Cache Compression

GLM-5.2 uses Multi-head Latent Attention (MLA), compressing KV caches from 32,768 floats per token to 576—a 57x reduction. colibrì persists the KV cache to disk (.coli_kv), so restarting continues conversations with zero re-prefill and byte-level consistency—critical when prefill is the slowest operation.

MTP Speculative Decoding: 2.2-2.8 Tokens per Forward

colibrì implements GLM-5.2's native Multi-Token Prediction (MTP) head for speculative decoding, achieving 2.2-2.8 tokens/forward—a huge win on a 0.3 tok/s system.

Hard-won lessons from the README:

> MTP head must be int8 (int4 heads collapse to 0-4% acceptance, #8)

Additionally, draft and verification must compute the same function—colibrì uses SPEC_PIN=1 to pin both to the same kernel family (see issue #163 for the full forensic report).

Honest Benchmarks

> This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan.

| Hardware | Speed | Notes | |---|---|---| | 6× RTX 5090 (fully resident) | 5.8-6.8 tok/s | TTFT ~13s | | 128 GB CPU desktop | ~1.8 tok/s | warm cache | | M5 Max (128 GB, Metal) | 1.06-1.83 tok/s | | | Ryzen AI 9 HX 370 (128 GB) | 0.37 tok/s | | | 25 GB machine | 0.05-0.1 tok/s | cold start, honest baseline |

Cloud H100 inference runs 30-50 tok/s—colibrì is 10-100x slower, but correct: forward passes are token-exact verified, matching the transformers reference implementation 32/32.

Why It Matters

Technical: colibrì redefines "loading a model"—from "read all weights into RAM" to "build an on-demand pipeline." Memory must only fit the activated parameters, not the whole model.
Industrial: GLM-5.2 is MIT-licensed open weights from Zhipu AI. Your code never leaves your machine, conversations aren't logged, and API bills are zero—significant for privacy-sensitive domains.
Philosophical: from the README: "Not renting intelligence behind an API — holding it." The web dashboard visualizes all 19,456 experts as a living cortex—color by storage tier, brightness by routing heat.

Three Engineering Principles

1. Know what the OS doesn't. Generic abstractions lose domain knowledge; bypassing them yields order-of-magnitude gains. 2. Measure honestly, don't assume quality. Every number cites its source; int4 quality loss and MTP acceptance rates are measured, not assumed. 3. Defaults are scar tissue. Both MTP rules exist because every new user would otherwise hit those pits.

The Deeper Insight: MoE Decouples Parameters from Memory

Dense models use 100% of parameters per token, so loading requires RAM ≥ model size. MoE models use only ~5.4%, meaning 94.6% of parameters are cold data at any moment—they can live on SSD, tape, or cloud storage. Deploying a 744B model now needs 25 GB RAM + 370 GB NVMe instead of 750 GB-1.5 TB of VRAM. Parameter count sets the intelligence ceiling; memory sets the deployment cost—and colibrì proves these can differ by 15x.

Roadmap: From "Runs" to "Usable"

Better prefetching (routing predictability from 71.6% toward 90%+)
GPU-accelerated matmul for dense layers (10-50x speedup potential)
Multithreaded expert loading to overlap I/O waits
Expert compression via structured pruning

Even as-is, colibrì proves one proposition: frontier models don't need frontier hardware—just frontier engineering.

---

Project: JustVugg/colibri (Apache 2.0) Model: GLM-5.2 (Zhipu AI, MIT license) Engine: pure C, zero dependencies, ~1,300 lines of core code Video: Devsplainers original video