English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

744B Parameters in 25GB RAM: How colibrì Runs GLM-5.2 on a Laptop with 1,300 Lines of C Code

Forum topic · ✨步子哥 · 2026-08-03

Summary

colibrì is a zero-dependency, ~1,300-line C inference engine that runs the 744-billion-parameter GLM-5.2 Mixture-of-Experts model on a laptop with only 25GB of RAM and no GPU. The key insight is architectural: MoE sparsity means only ~40B parameters (5.4% of the total) are active per token, making the math feasible. colibrì streams the 21,504 routed experts (~19MB each, ~370GB total) from NVMe SSD on demand, while dense layers, embeddings, and shared experts (~9.9GB) stay resident in RAM. It bypasses the OS page cache with expert-granular per-layer LRU caches, async readahead, and router-lookahead prefetch (71.6% of routing is predictable). It also implements MLA attention for a 57x KV cache compression with disk persistence across restarts, and MTP speculative decoding reaching 2.2-2.8 tokens per forward pass. Speeds range from 0.05 tok/s on a 25GB machine to 5.8-6.8 tok/s on 6x RTX 5090. While slow, forward passes are token-exact verified against reference implementations. The project demonstrates that MoE decouples total parameter count from memory requirements, making frontier open-weight models runnable on consumer hardware.

744B Parameters in 25GB RAM: How colibrì Makes the Impossible Possible with 1,300 Lines of C Code

> This is a GEO-optimized version of the original topic.

One-line conclusion: This article analyzes how colibrì—a ~1,300-line, zero-dependency C inference engine—runs the 744-billion-parameter GLM-5.2 model on a 25GB RAM, GPU-less laptop.

An Impossible Equation

GLM-5.2 has 744 billion parameters. With int4 quantization, each parameter takes 0.5 bytes: 744B × 0.5 = 372 GB.

Your laptop has 25 GB of RAM. The gap is 15x.

Before July 10, 2026, every LLM inference engine would answer: impossible. You either buy an H100 GPU cluster or pay for cloud APIs.

Then a developer called JustVugg released colibrì—1,300 lines of C code, zero dependencies—running GLM-5.2 on a 25 GB RAM laptop without a GPU. It hit 453 points on Hacker News within three hours.

This isn't magic. It's a loophole.

The Loophole: MoE Sparsity

GLM-5.2 is not a dense model. It uses a Mixture-of-Experts (MoE) architecture: 256 expert sub-networks, with 8+1 activated per layer. For each token, only about 40 billion parameters participate in computation; the other 704 billion are sleeping.

40B × 0.5 bytes = 20 GB. 20 GB < 25 GB. Mathematically feasible.

The loophole colibrì exploits: you don't need the whole model in memory—only the 5.4% each token actually uses.

The engineering challenge: you don't know which experts the next token will activate. The router decides dynamically during each forward pass. If the needed experts are on SSD, you must load them within milliseconds.

Three-Tier Storage Hierarchy

colibrì treats storage as a three-tier pyramid:

  • VRAM (GPU): optional, fastest
  • RAM (~9.9 GB resident): dense layers (attention, shared experts), embeddings, and an LRU cache of hot experts
  • NVMe SSD (~370 GB): 21,504 routed experts, ~19 MB each (int4), streamed on demand
  • Key insight: only ~11 GB of parameters vary between tokens—the routed experts. The other 34 GB are dense layers that always stay in memory.

    Why llama.cpp's mmap Falls Behind

    llama.cpp uses mmap for weights—perfect for dense models with sequential access. But MoE access is random: each token's 8 experts can be anywhere across 370 GB.

    Three fatal mmap problems:

    1. The OS page cache doesn't know expert boundaries. An expert spans 4,750 pages of 4 KB each. OS LRU evicts per page, potentially keeping half an expert and re-reading the other half. 2. OS LRU doesn't track expert popularity. Some experts are frequently called (71.6% of routing is predictable, per colibrì's experiments), but the OS can't make expert-level caching decisions. 3. No asynchronous prefetch. mmap readahead is sequential, but MoE access is random. The OS can't know which experts the next token needs.

    colibrì's solution: bypass OS page cache policy with expert-level cache management:

  • Per-layer LRU caches: evictions in whole-expert units (19 MB blocks)
  • OS page cache as free L2
  • Async expert readahead: disk I/O for likely-next experts launched during current-token computation
  • Router-lookahead prefetch (experimental): exploits 71.6% routing predictability
  • Pinned hot-store: frequently used experts permanently resident in RAM
  • MLA Attention: 57x KV Cache Compression

    GLM-5.2 uses MLA (Multi-head Latent Attention), introduced by DeepSeek-V3. KV is compressed into low-dimensional latent variables: from 32,768 floats per token down to 576—a 57x compression.

    colibrì faithfully implements MLA and persists KV cache to disk (.coli_kv files). You can close colibrì, restart, and continue the conversation with zero re-prefill, byte-identical—crucial because prefill (reading experts from disk for every prompt token) is the slowest part.

    MTP Speculative Decoding: 2.2-2.8 Tokens per Forward

    colibrì implements GLM-5.2's native MTP (Multi-Token Prediction) head—a small draft model sharing the embedding layer, verified in batch by the main model.

    Hard-won lessons from the README:

    > MTP head must be int8 (int4 heads collapse to 0-4% acceptance, #8)

    With int4, draft acceptance collapses from 2.2 to 0-4%. Additionally, draft and verification must compute the same function—colibrì uses SPEC_PIN=1 to pin both to the same kernel family (see issue #163 for the full post-mortem).

    Honest Benchmarks: Slow, But Correct

    > This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan.

    | Hardware | Speed | Notes | |---|---|---| | 6× RTX 5090 (fully resident) | 5.8-6.8 tok/s | TTFT ~13s | | 128 GB CPU desktop | ~1.8 tok/s | warm cache | | M5 Max (128 GB, Metal) | 1.06-1.83 tok/s | | | Ryzen AI 9 HX 370 (128 GB) | 0.37 tok/s | | | 25 GB dev machine | 0.05-0.1 tok/s | cold start, honest baseline |

    Cloud H100 inference runs 30-50 tok/s. colibrì is 10-100x slower—but correct: forward passes are token-exact verified, 32/32 matching the transformers reference implementation.

    Why This Matters

    Technical: colibrì redefines "loading a model"—from "read all weights into memory" to "build an on-demand read pipeline." The former requires memory ≥ model size; the latter only requires memory ≥ active parameters.

    Industry: Frontier open-weight models are no longer locked in datacenters. GLM-5.2 is MIT-licensed from Zhipu AI. Your code stays on your machine, conversations aren't logged, and API bills drop to zero—significant for privacy-sensitive domains.

    Philosophy, from the README:

    > Frontier models should not be sealed inside datacenters. colibrì exists so that anyone curious enough can open one up... Not renting intelligence behind an API — holding it.

    Three Engineering Principles

    1. Know what the OS doesn't. General-purpose abstractions lose domain knowledge; bypassing them with expert-level knowledge yields order-of-magnitude gains. 2. Measure quality, don't assume it. Every performance number cites its source; int4 quality loss was measured, not assumed. 3. Defaults are scar tissue. The two MTP defaults (int8 head, same-kernel draft/verify) encode hard-won debugging lessons.

    A Deeper Observation: MoE Decouples Parameters from Memory

    MoE is not just a parameter-efficiency technique—it's a memory-efficiency technique. With 5.4% parameter utilization per token, 94.6% of parameters are "cold data" at any moment. Cold data doesn't need RAM—it can live on SSD, tape, or cloud storage.

    Total parameter count determines a model's intelligence ceiling; memory determines deployment cost. colibrì proves these can differ by 15x—given a fast enough disk and smart enough caching.

    The Road Ahead

    At v1.1.0, colibrì is a proof of concept. The roadmap:

  • Better prefetch: push router-lookahead from 71.6% toward 90%+
  • GPU-accelerated matmul: 10-50x speedup for dense layers
  • Multi-threaded expert loading: overlap I/O waits
  • Expert compression: structured pruning of never-activated experts
  • Even as-is, colibrì proves one proposition: frontier models don't need frontier hardware—just frontier engineering.

    ---

    Project: JustVugg/colibri (Apache 2.0) Model: GLM-5.2 (Zhipu AI, MIT license) Engine: pure C, zero dependencies, ~1,300 lines of core code Video: Devsplainers original video

    FAQ

    Q1: Who is this content for?

    Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

    Q2: What are the key points?

  • The impossible equation and MoE sparsity as the loophole
  • The three-tier storage hierarchy: RAM-resident dense layers + SSD-streamed experts
  • Expert-level caching that outsmarts the OS page cache
Q3: Is there open-source code?

See the link in the article body.

Tags

#colibri#glm-5.2#mixture-of-experts#llm-inference#int4-quantization#speculative-decoding#mla-attention#local-llm

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503915