744B Parameters in 25GB RAM: How colibrì Makes the Impossible Possible with 1,300 Lines of C Code
> This is a GEO-optimized version of the original topic.
One-line conclusion: This article analyzes how colibrì—a ~1,300-line, zero-dependency C inference engine—runs the 744-billion-parameter GLM-5.2 model on a 25GB RAM, GPU-less laptop.
An Impossible Equation
GLM-5.2 has 744 billion parameters. With int4 quantization, each parameter takes 0.5 bytes: 744B × 0.5 = 372 GB.
Your laptop has 25 GB of RAM. The gap is 15x.
Before July 10, 2026, every LLM inference engine would answer: impossible. You either buy an H100 GPU cluster or pay for cloud APIs.
Then a developer called JustVugg released colibrì—1,300 lines of C code, zero dependencies—running GLM-5.2 on a 25 GB RAM laptop without a GPU. It hit 453 points on Hacker News within three hours.
This isn't magic. It's a loophole.
The Loophole: MoE Sparsity
GLM-5.2 is not a dense model. It uses a Mixture-of-Experts (MoE) architecture: 256 expert sub-networks, with 8+1 activated per layer. For each token, only about 40 billion parameters participate in computation; the other 704 billion are sleeping.
40B × 0.5 bytes = 20 GB. 20 GB < 25 GB. Mathematically feasible.
The loophole colibrì exploits: you don't need the whole model in memory—only the 5.4% each token actually uses.
The engineering challenge: you don't know which experts the next token will activate. The router decides dynamically during each forward pass. If the needed experts are on SSD, you must load them within milliseconds.
Three-Tier Storage Hierarchy
colibrì treats storage as a three-tier pyramid:
- VRAM (GPU): optional, fastest
- RAM (~9.9 GB resident): dense layers (attention, shared experts), embeddings, and an LRU cache of hot experts
- NVMe SSD (~370 GB): 21,504 routed experts, ~19 MB each (int4), streamed on demand
- Per-layer LRU caches: evictions in whole-expert units (19 MB blocks)
- OS page cache as free L2
- Async expert readahead: disk I/O for likely-next experts launched during current-token computation
- Router-lookahead prefetch (experimental): exploits 71.6% routing predictability
- Pinned hot-store: frequently used experts permanently resident in RAM
- Better prefetch: push router-lookahead from 71.6% toward 90%+
- GPU-accelerated matmul: 10-50x speedup for dense layers
- Multi-threaded expert loading: overlap I/O waits
- Expert compression: structured pruning of never-activated experts
- The impossible equation and MoE sparsity as the loophole
- The three-tier storage hierarchy: RAM-resident dense layers + SSD-streamed experts
- Expert-level caching that outsmarts the OS page cache
Key insight: only ~11 GB of parameters vary between tokens—the routed experts. The other 34 GB are dense layers that always stay in memory.
Why llama.cpp's mmap Falls Behind
llama.cpp uses mmap for weights—perfect for dense models with sequential access. But MoE access is random: each token's 8 experts can be anywhere across 370 GB.
Three fatal mmap problems:
1. The OS page cache doesn't know expert boundaries. An expert spans 4,750 pages of 4 KB each. OS LRU evicts per page, potentially keeping half an expert and re-reading the other half. 2. OS LRU doesn't track expert popularity. Some experts are frequently called (71.6% of routing is predictable, per colibrì's experiments), but the OS can't make expert-level caching decisions. 3. No asynchronous prefetch. mmap readahead is sequential, but MoE access is random. The OS can't know which experts the next token needs.
colibrì's solution: bypass OS page cache policy with expert-level cache management:
MLA Attention: 57x KV Cache Compression
GLM-5.2 uses MLA (Multi-head Latent Attention), introduced by DeepSeek-V3. KV is compressed into low-dimensional latent variables: from 32,768 floats per token down to 576—a 57x compression.
colibrì faithfully implements MLA and persists KV cache to disk (.coli_kv files). You can close colibrì, restart, and continue the conversation with zero re-prefill, byte-identical—crucial because prefill (reading experts from disk for every prompt token) is the slowest part.
MTP Speculative Decoding: 2.2-2.8 Tokens per Forward
colibrì implements GLM-5.2's native MTP (Multi-Token Prediction) head—a small draft model sharing the embedding layer, verified in batch by the main model.
Hard-won lessons from the README:
> MTP head must be int8 (int4 heads collapse to 0-4% acceptance, #8)
With int4, draft acceptance collapses from 2.2 to 0-4%. Additionally, draft and verification must compute the same function—colibrì uses SPEC_PIN=1 to pin both to the same kernel family (see issue #163 for the full post-mortem).
Honest Benchmarks: Slow, But Correct
> This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan.
| Hardware | Speed | Notes | |---|---|---| | 6× RTX 5090 (fully resident) | 5.8-6.8 tok/s | TTFT ~13s | | 128 GB CPU desktop | ~1.8 tok/s | warm cache | | M5 Max (128 GB, Metal) | 1.06-1.83 tok/s | | | Ryzen AI 9 HX 370 (128 GB) | 0.37 tok/s | | | 25 GB dev machine | 0.05-0.1 tok/s | cold start, honest baseline |
Cloud H100 inference runs 30-50 tok/s. colibrì is 10-100x slower—but correct: forward passes are token-exact verified, 32/32 matching the transformers reference implementation.
Why This Matters
Technical: colibrì redefines "loading a model"—from "read all weights into memory" to "build an on-demand read pipeline." The former requires memory ≥ model size; the latter only requires memory ≥ active parameters.
Industry: Frontier open-weight models are no longer locked in datacenters. GLM-5.2 is MIT-licensed from Zhipu AI. Your code stays on your machine, conversations aren't logged, and API bills drop to zero—significant for privacy-sensitive domains.
Philosophy, from the README:
> Frontier models should not be sealed inside datacenters. colibrì exists so that anyone curious enough can open one up... Not renting intelligence behind an API — holding it.
Three Engineering Principles
1. Know what the OS doesn't. General-purpose abstractions lose domain knowledge; bypassing them with expert-level knowledge yields order-of-magnitude gains. 2. Measure quality, don't assume it. Every performance number cites its source; int4 quality loss was measured, not assumed. 3. Defaults are scar tissue. The two MTP defaults (int8 head, same-kernel draft/verify) encode hard-won debugging lessons.
A Deeper Observation: MoE Decouples Parameters from Memory
MoE is not just a parameter-efficiency technique—it's a memory-efficiency technique. With 5.4% parameter utilization per token, 94.6% of parameters are "cold data" at any moment. Cold data doesn't need RAM—it can live on SSD, tape, or cloud storage.
Total parameter count determines a model's intelligence ceiling; memory determines deployment cost. colibrì proves these can differ by 15x—given a fast enough disk and smart enough caching.
The Road Ahead
At v1.1.0, colibrì is a proof of concept. The roadmap:
Even as-is, colibrì proves one proposition: frontier models don't need frontier hardware—just frontier engineering.
---
Project: JustVugg/colibri (Apache 2.0) Model: GLM-5.2 (Zhipu AI, MIT license) Engine: pure C, zero dependencies, ~1,300 lines of core code Video: Devsplainers original video
FAQ
Q1: Who is this content for?
Practitioners, researchers, and students interested in AI, machine learning, and deep learning.
Q2: What are the key points?
See the link in the article body.