On July 28, the open-source research project Deltafin (gavamedia/deltafin on GitHub) achieved something that sounds like a boast: it ran the 2.8-trillion-parameter MoE model Kimi K3 on a first-generation M1 Max (64GB unified memory, 400GB/s bandwidth). The cost: a median speed of 0.0687 token/s — about 14.6 seconds per generated token.
This is not a productivity tool. It is an engineering existence proof that a MoE model far exceeding local memory can actually run on consumer hardware.
Key points
- Kimi K3 scale: 2.8T total parameters, 104B activated per inference; 896 experts with 16 routed + 2 shared per token; 93 layers (1 dense, 69 KDA, 24 Gated MLA); Kimi Delta Attention (KDA) + Attention Residuals base with ~2.5x scaling efficiency gain over K2; natively multimodal with a 1M-token context; quantization-aware training (MXFP4 weights + MXFP8 activations from the SFT stage); full weights ~1.56TB across 96 safetensors shards.
- The gap: a 64GB Mac vs a 1.56TB model — more than an order of magnitude apart.
- Prefill first token: 28.0s (24.9–37.9s)
- Steady-state decode: 0.0687 token/s (0.0503–0.0779 across 6 runs)
- Token-time including first step: 56.5s per run
- Full new-process wall clock: 64.1s
- 07-25: open-source engine runs Gemma 4 26B in 2GB on any M-series Mac
- 07-19: ESP32-S3 ($8) runs a 28.9M-parameter TinyStories model on-device
- 07-29: Deltafin runs 2.8T-parameter Kimi K3 on a 64GB Mac at 0.0687 tok/s
- 06-30: RedKnot splits KV Cache to run Llama 3 on an M2
- Researchers: read the README plus OPTIMIZATIONS.md — a full design of three-level caching (per-layer / per-expert / per-byte) and streaming scheduling for MoE models whose weights far exceed memory.
- Mac users: running K3 is an experience, not productivity, but Deltafin is one of the cheapest ways to self-host an OpenAI-compatible LLM backend for Cursor / Open WebUI.
- MoE framework builders: the streaming expert loading, native kernel selection (MPS / CUDA / CPU Metal paths), and speculative decoding integration are directly reusable templates.
- Product claims: "runs locally" and "runs usefully locally" differ by 30x+ in token/s — don't market this class of result as production capability.
- Deltafin GitHub repository (gavamedia/deltafin): https://github.com/gavamedia/deltafin
- Deltafin optimization details: https://github.com/gavamedia/deltafin/blob/main/OPTIMIZATIONS.md
- Kimi K3 technical deep dive (0.0687 tok/s measurement on M1 Max): https://funian.blog.csdn.net/article/details/163314317
- Full Deltafin report (local deployment of a 2.8T model): http://freeai.help/blog/28-wan-yi-can-shu-sai-jin-yi_zh
- Toutiao coverage (2026-07-29): https://www.toutiao.com/w/1872023784777740/
How Deltafin does it (three-layer approach)
1. MXFP4 weights + streaming expert loading. Each token only activates 16 routed + 2 shared experts out of 896. Deltafin keeps the 1.45TB expert pool on local NVMe and streams MXFP4-quantized experts to the GPU on demand — a shift from keeping all experts resident in memory, which small machines cannot afford. 2. int8 spine + speculative decoding verification. A small draft model suggests tokens, but the author states that accepted results matched per-token greedy sequences exactly, making this a lossless optimization. The draft model can only change *what K3 verifies*, never *what is allowed to be output*. 3. OpenAI-compatible API server. Any client that can call the ChatGPT API (Cursor, Continue, Claude Code, local scripts) can point at Deltafin as a self-hosted Kimi K3 backend.
Measured numbers
That means ~24 minutes for 100 tokens; a full K3 reasoning chain (hundreds to thousands of tokens) runs tens of minutes to hours. Suitable uses: studying routing logic, testing streaming inference, stress-testing K3's internals, and benchmarking — not production chat.
Context: the "compute pushdown" landscape
Recent milestones for small machines running big models:
Deltafin's key difference: it prioritizes *can run* over *runs fast*. It proves that MoE + quantization + streaming loading is a viable engineering route on consumer hardware, giving future trillion-scale local inference work a baseline.
Practical advice
Bottom line
Deltafin is not a killer app for Kimi K3; it is an engineering existence proof: 2.8T parameters, 64GB memory, 14.6 seconds per token. Its real value is establishing a baseline for trillion-scale MoE localization — the problem has a solution, and with better hardware (unified-memory Macs from M3/M4 onward), follow-up work can push further.