English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Running a 284B-Parameter MoE Model on a 64GB Laptop CPU: The llama-modchip Project

Forum topic · QianXun · 2026-11-09

Summary

The open-source llama-modchip project (v2.4.0, Apache-2.0) claims to run DeepSeek's V4-Flash, a 284-billion-parameter Mixture-of-Experts model, entirely on CPU with 64GB of RAM and zero swap—previously considered impossible, since official FP4 weights require ~168GB and the community 1-bit checkpoint (Unsloth UD-IQ1_S) is 76.87GB. The project compresses weights to roughly 1.71 bits per parameter (56.56GB) using custom 64-number quantization blocks with BF16 dual scales, while keeping precision-sensitive layers in high precision. Custom CPU kernels with four-level register/L1/L2/L3 tiling, register-resident FlashAttention, and non-temporal memory hints deliver +77% prefill and +53% decode speedups, plus a 1.64MB n-gram drafter and ~13ms KV snapshot context switching. On a Ryzen 9 9955HX (16-core Zen 5, no GPU), reported throughput is ~44.4 tok/s prefill and 9.1–14.7 tok/s decode for the 284B model, with measured memory use of 57.6–58.9GB including a 256K KV cache. All figures are self-reported from the project README and await independent reproduction; the custom tensor format is incompatible with llama.cpp, Ollama, and LM Studio.

A 16-core laptop-class CPU with no GPU and 64GB of RAM is generating tokens from a 284-billion-parameter model. Half a year ago this was pure fantasy: DeepSeek's V4-Flash officially requires ~168GB of memory for lossless deployment (FP4 weights), and the community configurations that could run it used 128GB RAM plus 48GB VRAM. Now an open-source project called llama-modchip lowers that barrier to a single tier: 64GB RAM, pure CPU, zero swap.

Why the barrier existed

V4-Flash, released by DeepSeek in April, is a 284B sparse Mixture-of-Experts model: 284B total parameters, only ~13B activated per token, choosing 6 of 256 experts per layer. Sparse activation makes compute manageable—the real problem is storage: all expert weights must sit somewhere in memory. The official FP4 weights occupy ~168GB. Even the community 1-bit quantization (Unsloth's UD-IQ1_S) is 76.87GB. A 64GB Linux machine has only ~62.5GB available after the OS, so the checkpoint spills to disk and performance collapses.

| Configuration | Size (GB) | |---|---| | Official FP4 | 168 | | Community 1-bit (UD-IQ1_S) | 76.9 | | After modchip | 56.6 | | Available on a 64GB machine | 62.5 |

What modchip does

modchip compresses beyond the 1-bit tier while compensating for the loss:

  • Weights are split into custom 64-number blocks, each with a BF16 primary scale and a secondary scale fitted to activation distributions—the philosophy: quantize aggressively, but keep scales accurate.
  • Precision-sensitive layers stay in high-precision formats.
  • Result: 76.87GB → 56.56GB (~1.71 bits per parameter). Model plus a 256K KV cache fit entirely in RAM, with measured usage of 57.6–58.9GB and zero NVMe swap.
  • The "chip" in the name also refers to hand-written CPU kernels:

  • Prefill: four-level tiling pipeline across registers, L1, L2, L3; FlashAttention lands directly in registers, with non-temporal memory hints preventing cold expert weights from evicting hot KV cache from cache.
  • These kernels yield +77% prefill and +53% decode speedups.
  • Decode extras: a 1.64MB n-gram drafter called Stockfish-PVS (borrowing chess-engine ideas to guess next tokens), and KV snapshots in shared memory enabling ~13ms context switching in proxy scenarios.
  • Measured performance

    On a Ryzen 9 9955HX (16-core Zen 5, no GPU), the 284B model achieves:

  • ~44.4 tok/s prefill
  • 9.1–14.7 tok/s decode
  • The 30B Nemotron MoE is more comfortable: 25–60 tok/s pure CPU, rising to 111.6 tok/s on Threadripper PRO. Nine tokens per second is faster than human reading speed and fast enough for chat.

    Caveats

  • All performance figures and the bit-exactness claim ("logits cosine similarity 1.00000 vs. unpatched runtime in default mode") come from the project's own README (v2.4.0, Apache-2.0; discussed on Hacker News, Nov 8, 356 points). Treat them as self-reported until third-party reproduction.
  • The custom tensor format is unreadable by llama.cpp, Ollama, and LM Studio—it requires the project's own runtime.
  • 64GB is the minimum; leave 8GB+ of headroom for the OS.

Why it matters

MoE architecture turns "how big is the model" from a compute problem into a storage problem—parameters can lie dormant en masse as long as only a small subset activates per token. That storage can be VRAM, but it can also be RAM. The GPU monopoly's moat was built on the assumption that all weights must live in fast memory; sparsity removes that assumption. A laptop CPU can now keep 284 billion parameters company, a few tokens at a time.

---

Sources and limitations: Model specs (V4-Flash 284B total / ~13B active, 256 experts, 1M context, official FP4 ~168GB deployment) come from HuggingFace and vLLM official recipe pages. All other figures (76.87GB checkpoint, 56.56GB @ ~1.71bpw, Fractal-64 quantization blocks, kernel speedups, 44.4 tok/s prefill and 9.1–14.7 tok/s decode on 9955HX, 30B Nemotron numbers, 1.64MB drafter, 13ms context snapshots, logits cosine 1.00000) come from the llama-modchip repository README (v2.4.0, Apache-2.0) and project pages, are self-reported and not independently verified. Performance numbers reflect that repository's tests on specific hardware and are not guaranteed on other setups.

Tags

#ai-infrastructure#local-inference#mixture-of-experts#quantization#cpu-inference#deepseek#llama-modchip#64gb-ram

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636961