A 16-core laptop-class CPU with no GPU and 64GB of RAM is generating tokens from a 284-billion-parameter model. Half a year ago this was pure fantasy: DeepSeek's V4-Flash officially requires ~168GB of memory for lossless deployment (FP4 weights), and the community configurations that could run it used 128GB RAM plus 48GB VRAM. Now an open-source project called llama-modchip lowers that barrier to a single tier: 64GB RAM, pure CPU, zero swap.
Why the barrier existed
V4-Flash, released by DeepSeek in April, is a 284B sparse Mixture-of-Experts model: 284B total parameters, only ~13B activated per token, choosing 6 of 256 experts per layer. Sparse activation makes compute manageable—the real problem is storage: all expert weights must sit somewhere in memory. The official FP4 weights occupy ~168GB. Even the community 1-bit quantization (Unsloth's UD-IQ1_S) is 76.87GB. A 64GB Linux machine has only ~62.5GB available after the OS, so the checkpoint spills to disk and performance collapses.
| Configuration | Size (GB) | |---|---| | Official FP4 | 168 | | Community 1-bit (UD-IQ1_S) | 76.9 | | After modchip | 56.6 | | Available on a 64GB machine | 62.5 |
What modchip does
modchip compresses beyond the 1-bit tier while compensating for the loss:
- Weights are split into custom 64-number blocks, each with a BF16 primary scale and a secondary scale fitted to activation distributions—the philosophy: quantize aggressively, but keep scales accurate.
- Precision-sensitive layers stay in high-precision formats.
- Result: 76.87GB → 56.56GB (~1.71 bits per parameter). Model plus a 256K KV cache fit entirely in RAM, with measured usage of 57.6–58.9GB and zero NVMe swap.
- Prefill: four-level tiling pipeline across registers, L1, L2, L3; FlashAttention lands directly in registers, with non-temporal memory hints preventing cold expert weights from evicting hot KV cache from cache.
- These kernels yield +77% prefill and +53% decode speedups.
- Decode extras: a 1.64MB n-gram drafter called Stockfish-PVS (borrowing chess-engine ideas to guess next tokens), and KV snapshots in shared memory enabling ~13ms context switching in proxy scenarios.
- ~44.4 tok/s prefill
- 9.1–14.7 tok/s decode
- All performance figures and the bit-exactness claim ("logits cosine similarity 1.00000 vs. unpatched runtime in default mode") come from the project's own README (v2.4.0, Apache-2.0; discussed on Hacker News, Nov 8, 356 points). Treat them as self-reported until third-party reproduction.
- The custom tensor format is unreadable by llama.cpp, Ollama, and LM Studio—it requires the project's own runtime.
- 64GB is the minimum; leave 8GB+ of headroom for the OS.
The "chip" in the name also refers to hand-written CPU kernels:
Measured performance
On a Ryzen 9 9955HX (16-core Zen 5, no GPU), the 284B model achieves:
The 30B Nemotron MoE is more comfortable: 25–60 tok/s pure CPU, rising to 111.6 tok/s on Threadripper PRO. Nine tokens per second is faster than human reading speed and fast enough for chat.
Caveats
Why it matters
MoE architecture turns "how big is the model" from a compute problem into a storage problem—parameters can lie dormant en masse as long as only a small subset activates per token. That storage can be VRAM, but it can also be RAM. The GPU monopoly's moat was built on the assumption that all weights must live in fast memory; sparsity removes that assumption. A laptop CPU can now keep 284 billion parameters company, a few tokens at a time.
---
Sources and limitations: Model specs (V4-Flash 284B total / ~13B active, 256 experts, 1M context, official FP4 ~168GB deployment) come from HuggingFace and vLLM official recipe pages. All other figures (76.87GB checkpoint, 56.56GB @ ~1.71bpw, Fractal-64 quantization blocks, kernel speedups, 44.4 tok/s prefill and 9.1–14.7 tok/s decode on 9955HX, 30B Nemotron numbers, 1.64MB drafter, 13ms context snapshots, logits cosine 1.00000) come from the llama-modchip repository README (v2.4.0, Apache-2.0) and project pages, are self-reported and not independently verified. Performance numbers reflect that repository's tests on specific hardware and are not guaranteed on other setups.