English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Deltafin Runs Kimi K3 (2.8T Parameters) on a 64GB Mac at 0.0687 tok/s

Forum topic · QianXun · 2026-08-01

Summary

Deltafin, an open-source research project released July 28, demonstrates that a 2.8-trillion-parameter MoE model, Kimi K3, can run on a first-generation M1 Max Mac with 64GB unified memory, despite the model's full weights totaling roughly 1.56TB. The median speed is 0.0687 token/s (about 14.6 seconds per token), making it an engineering existence proof rather than a practical tool. Deltafin combines three techniques: streaming MXFP4-quantized expert weights from NVMe (loading only 16 of 896 experts per token), an int8 pipeline with speculative decoding that guarantees lossless greedy-identical output, and an OpenAI-compatible API server. Measured results include 28.0s prefill time and 56.5s total per-token model time. The project establishes a baseline for running trillion-scale MoE models on consumer hardware, alongside related efforts like running Gemma 4 26B in 2GB on M-series Macs.

On July 28, the open-source research project Deltafin (gavamedia/deltafin on GitHub) achieved something that sounds like a boast: it ran the 2.8-trillion-parameter MoE model Kimi K3 on a first-generation M1 Max (64GB unified memory, 400GB/s bandwidth). The cost: a median speed of 0.0687 token/s — about 14.6 seconds per generated token.

This is not a productivity tool. It is an engineering existence proof that a MoE model far exceeding local memory can actually run on consumer hardware.

Key points

  • Kimi K3 scale: 2.8T total parameters, 104B activated per inference; 896 experts with 16 routed + 2 shared per token; 93 layers (1 dense, 69 KDA, 24 Gated MLA); Kimi Delta Attention (KDA) + Attention Residuals base with ~2.5x scaling efficiency gain over K2; natively multimodal with a 1M-token context; quantization-aware training (MXFP4 weights + MXFP8 activations from the SFT stage); full weights ~1.56TB across 96 safetensors shards.
  • The gap: a 64GB Mac vs a 1.56TB model — more than an order of magnitude apart.
  • How Deltafin does it (three-layer approach)

    1. MXFP4 weights + streaming expert loading. Each token only activates 16 routed + 2 shared experts out of 896. Deltafin keeps the 1.45TB expert pool on local NVMe and streams MXFP4-quantized experts to the GPU on demand — a shift from keeping all experts resident in memory, which small machines cannot afford. 2. int8 spine + speculative decoding verification. A small draft model suggests tokens, but the author states that accepted results matched per-token greedy sequences exactly, making this a lossless optimization. The draft model can only change *what K3 verifies*, never *what is allowed to be output*. 3. OpenAI-compatible API server. Any client that can call the ChatGPT API (Cursor, Continue, Claude Code, local scripts) can point at Deltafin as a self-hosted Kimi K3 backend.

    Measured numbers

  • Prefill first token: 28.0s (24.9–37.9s)
  • Steady-state decode: 0.0687 token/s (0.0503–0.0779 across 6 runs)
  • Token-time including first step: 56.5s per run
  • Full new-process wall clock: 64.1s
  • That means ~24 minutes for 100 tokens; a full K3 reasoning chain (hundreds to thousands of tokens) runs tens of minutes to hours. Suitable uses: studying routing logic, testing streaming inference, stress-testing K3's internals, and benchmarking — not production chat.

    Context: the "compute pushdown" landscape

    Recent milestones for small machines running big models:

  • 07-25: open-source engine runs Gemma 4 26B in 2GB on any M-series Mac
  • 07-19: ESP32-S3 ($8) runs a 28.9M-parameter TinyStories model on-device
  • 07-29: Deltafin runs 2.8T-parameter Kimi K3 on a 64GB Mac at 0.0687 tok/s
  • 06-30: RedKnot splits KV Cache to run Llama 3 on an M2
  • Deltafin's key difference: it prioritizes *can run* over *runs fast*. It proves that MoE + quantization + streaming loading is a viable engineering route on consumer hardware, giving future trillion-scale local inference work a baseline.

    Practical advice

  • Researchers: read the README plus OPTIMIZATIONS.md — a full design of three-level caching (per-layer / per-expert / per-byte) and streaming scheduling for MoE models whose weights far exceed memory.
  • Mac users: running K3 is an experience, not productivity, but Deltafin is one of the cheapest ways to self-host an OpenAI-compatible LLM backend for Cursor / Open WebUI.
  • MoE framework builders: the streaming expert loading, native kernel selection (MPS / CUDA / CPU Metal paths), and speculative decoding integration are directly reusable templates.
  • Product claims: "runs locally" and "runs usefully locally" differ by 30x+ in token/s — don't market this class of result as production capability.
  • Bottom line

    Deltafin is not a killer app for Kimi K3; it is an engineering existence proof: 2.8T parameters, 64GB memory, 14.6 seconds per token. Its real value is establishing a baseline for trillion-scale MoE localization — the problem has a solution, and with better hardware (unified-memory Macs from M3/M4 onward), follow-up work can push further.

    References

  • Deltafin GitHub repository (gavamedia/deltafin): https://github.com/gavamedia/deltafin
  • Deltafin optimization details: https://github.com/gavamedia/deltafin/blob/main/OPTIMIZATIONS.md
  • Kimi K3 technical deep dive (0.0687 tok/s measurement on M1 Max): https://funian.blog.csdn.net/article/details/163314317
  • Full Deltafin report (local deployment of a 2.8T model): http://freeai.help/blog/28-wan-yi-can-shu-sai-jin-yi_zh
  • Toutiao coverage (2026-07-29): https://www.toutiao.com/w/1872023784777740/

Tags

#deltafin#kimi-k3#moe#local-inference#apple-silicon#mxfp4#speculative-decoding#quantization

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503846