Colibri: Running a 744B MoE Model on 16GB RAM in Pure C
> Project: JustVugg/colibri > Stars: 960 (first trending 2026-09-13) > Language: C > License: MIT
The Scenario
You want to run GLM-5.2—a 744B-parameter MoE model—on your laptop. The traditional answer: you need 8x H100, roughly 640GB of VRAM. You don't have 8 H100s; you have a MacBook with 16GB of RAM.
Colibri says that's enough.
It keeps the dense parts of the 744B model (attention, shared experts, embedding layers) resident in 16GB of RAM, and places the routed experts on disk, reading them on demand. Each generated token activates only ~40B parameters, of which only ~11GB changes between tokens—this is what gets read from disk.
Result: a 744B model runs on your laptop. Not the fastest, but it runs.
Core Insight: Huge on Disk, Small Per Token
Colibri's design rests on one observation: MoE models are enormous on disk but small per token.
GLM-5.2 has 744B parameters, but each token uses only ~40B. Of that, ~29B is the dense part (used by every token) and ~11B is routed experts (different tokens use different experts).
This means you don't need all 744B in memory. Keep the 29B dense part in RAM; read the ~11GB of varying experts from disk on demand. 11GB of random reads takes roughly 1-2 seconds on an NVMe SSD—not fast enough for real-time interaction, but sufficient for batch inference.
Colibri's core approach: place the model on the appropriate storage tiers instead of forcing it into one.
Three-Tier Storage Architecture
RAM (fastest): the dense parts—attention weights, shared experts, embeddings, LayerNorm. Used every token, so they must be fastest. GLM-5.2's dense part is ~29B parameters, ~15GB after int4 quantization—just fits in 16GB.
GPU VRAM (if available): the hottest experts. Colibri tracks which experts are called frequently and caches them on GPU. Without a GPU, this tier is empty and everything runs on RAM + disk.
Disk (slowest, largest): all routed experts. A 744B model's experts total 700GB+, stored on disk. Colibri manages them with a learning cache that records your usage patterns and prefetches experts you'll likely need.
This architecture resembles OS virtual memory paging strikingly—hot data in RAM, cold data on disk, paged on demand. The difference: Colibri swaps expert modules rather than memory pages.
The Learning Cache: Not LRU, but Understanding You
Colibri's disk cache is not simple LRU. It's a learning cache—it records your usage patterns and predicts which experts you'll need next.
Why? Expert invocation has locality: code-related experts fire frequently during programming tasks; logic-related experts during math reasoning. If you're doing similar tasks, many experts get reused.
The cache learns this pattern. Over time, hit rates rise, disk reads fall, inference speeds up—like a CPU branch predictor: correct predictions are fast, wrong ones slow, but predicting beats not predicting.
Pure C, Zero Dependencies, One File Per Model Family
Colibri's most surprising choice: pure C, zero external dependencies, one .c file per model family.
Not nostalgia—engineering. The author's reasons:
1. Measurability: C's behavior is transparent—no hidden allocations, no GC pauses, no JIT compile latency. What you measure is what you get. You can't optimize what you can't see.
2. Portability: a C compiler exists on every platform—Windows, Linux, macOS, even Raspberry Pi. Zero dependencies means no CUDA toolkit, no Python environment, no vLLM dependency tree.
3. Single-file architecture: one .c file per model family. GLM-5.2's inference logic lives in one file, DeepSeek V4 Flash in another. Want to understand how a model runs? Read one file. Want to optimize it? Change one file.
This follows the llama.cpp philosophy—C/C++ single binaries vs. Python dependency hell—but Colibri is more radical: pure C vs. C++, and zero dependencies even against the standard library.
The "Placement" Philosophy
Colibri repeatedly stresses one concept: placement.
Traditional inference frameworks think in terms of "fitting"—however big the model, you need that much memory/VRAM. Doesn't fit? Buy bigger cards.
Colibri thinks in terms of placement—put different parts of the model on different storage tiers according to access frequency and latency needs. Dense parts used every token go in RAM; routed experts go on disk; hottest experts go on GPU.
The idea isn't new—OS storage hierarchies (L1/L2/L3 cache → RAM → SSD → HDD) work exactly this way. But Colibri is the first open-source project to apply it systematically to MoE inference.
Key design principle: "where weights live affects speed, not results." Moving experts from GPU to disk makes inference slower but changes nothing about the output. Colibri's defaults never silently alter model precision or routing semantics—slowness is acceptable; wrong results are not.
13 Engines, One Framework
Colibri currently supports 13 engines:
10 language models: GLM-5.2/5.3, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, DeepSeek V4.1 Flash, MiMo-V2.6 Flash/Pro, Qwen3.8-Flash-Next, Qwen3.6 (compatible with Qwen3-Coder and the dense Qwen3.8-27B), OLMoE
1 image model: Qwen-Image-2.1
2 decision models: Laya, GLiNER2.5-Decide (a third, Clef, runs on the Qwen3.6 engine)
Each engine is one .c file. Adding a model family = adding a .c file. The framework core (cache, routing, API server) is shared. This lets Colibri support new models quickly—when a new MoE model ships, you write a new engine file without touching the framework, similar to PyTorch's custom-op registration but far lighter.
OpenAI/Anthropic API Compatible
Colibri exposes two API endpoints once running:
- OpenAI-compatible:
http://127.0.0.1:8000/v1 - Anthropic-compatible:
http://127.0.0.1:8000
MCP Server
Colibri also ships an MCP server (coli mcp) letting AI assistants manage Colibri automatically: detect hardware, recommend models, install, start, stop, check status.
This means AI agents can deploy AI inference infrastructure themselves—"install me a local LLM," and the agent handles hardware detection, model selection, installation, and startup via MCP. An instance of "AI managing AI infrastructure."
One-Line Summary
Colibri turns "a 744B MoE model needs 8x H100" into "needs 16GB RAM + 22GB disk." Not by compressing the model, but by rethinking where the model lives across the storage hierarchy—dense parts resident in memory, experts read from disk on demand. The pure C, zero-dependency, one-file-per-family implementation makes this placement philosophy transparent and measurable.