Overview
Deltafin, released on July 28, is an open-source research project that demonstrates running Kimi K3, a 2.8-trillion-parameter MoE language model, on a base M1 Max Mac with 64GB of unified memory. Median inference speed measured 0.0687 tokens per second—about 14.6 seconds per token—positioning Deltafin as an engineering existence proof rather than a production-ready productivity tool. It shows that MoE models vastly exceeding local memory can in fact run on consumer hardware.
Kimi K3 Scale
Kimi K3, open-sourced by Moonshot in July, has the following architecture, per its Hugging Face model card:
- 2.8T total parameters with 104B activated per inference
- 896 experts; 16 selected per token via the router, plus 2 shared experts
- 93 layers (1 dense layer + 69 KDA layers + 24 Gated MLA layers)
- Architecture built on Kimi Delta Attention (KDA) + Attention Residuals, delivering roughly 2.5× improvement in scaling efficiency over K2
- Native multimodal, 1M-token context window
- Quantization-aware training, with MXFP4 weights and MXFP8 activations beginning at the SFT stage
- Full weights are approximately 1.56TB across 96 safetensors shards
- Prefill first token: 28.0 seconds (range 24.9–37.9s)
- Steady-state decoding: 0.0687 tok/s (6 runs, range 0.0503–0.0779)
- Per-token model time including first step: 56.5 seconds across 6 runs
- Wall-clock for a fresh process: 64.1 seconds
- 07-25: An open-source engine runs Gemma 4 26B on any M-series Mac with 2GB of memory
- 07-19: A $8 ESP32-S3 runs TinyStories at 28.9M parameters on-device
- 07-29: Deltafin runs Kimi K3 at 2.8T parameters on a 64GB Mac at 0.0687 tok/s
- 06-30: RedKnot splits KV cache per head to run Llama 3 on an M2
- Researchers: Deltafin is a reference architecture for "how to run an MoE model when memory is much smaller than the weights." The design is not "load it and done" but a complete three-tier caching and streaming scheduler across layers, experts, and bytes. Read the README alongside OPTIMIZATIONS.md.
- Mac users: Running Kimi K3 via Deltafin is an experience, not productivity. However, it offers one of the cheapest open-source implementations of a local OpenAI-compatible LLM backend; clients like Cursor or Open WebUI can point at it.
- MoE framework developers: Deltafin's streaming-expert logic, native-kernel selection across MPS / CUDA / CPU Metal paths, and speculative-decoding integration are reusable engineering templates.
- Products claiming "local model X": Deltafin illustrates that "can run" and "can run usefully" differ by more than 30× in tokens per second. The category is still pushing the ceiling, not delivering production-grade performance.
- Deltafin GitHub repository: https://github.com/gavamedia/deltafin
- Deltafin optimization mechanisms (OPTIMIZATIONS.md): https://github.com/gavamedia/deltafin/blob/main/OPTIMIZATIONS.md
- Kimi K3 deep technical analysis (M1 Max deployment, 0.0687 tok/s measurement): https://funian.blog.csdn.net/article/details/163314317
- Deltafin Chinese/English coverage: http://freeai.help/blog/28-wan-yi-can-shu-sai-jin-yi_zh
- Deltafin Toutiao coverage: https://www.toutiao.com/w/1872023784777740/
A 64GB Mac against a 1.56TB model represents more than an order-of-magnitude gap.
Deltafin's Three-Layer Approach
Deltafin does not rely on a single trick. Its README describes a three-layer combination:
Layer 1 — MXFP4 Weights with Streaming Expert Loading
Each token activates only 16 experts plus 2 shared experts, not all 896. Deltafin keeps the 1.45TB expert library on local NVMe storage and streams the relevant portions to the GPU on demand in MXFP4 format. This is the first practical "read-on-demand" pattern for MoE inference, replacing the prior assumption that all experts must reside in VRAM or host RAM.
Layer 2 — int8 Speculative Decoding Spine
Deltafin uses a small model as a drafter. The authors explicitly state that observed acceptance results match a greedy token-by-token sequence exactly, making this a lossless optimization. The small model only proposes work; its confidence and selection only change *what K3 verifies*, not *what is allowed to be output*. Under this engineering discipline, the small model functions more like a pre-fetcher and cache warmer than an answer generator for K3.
Layer 3 — OpenAI-Compatible API Server
Once running, Deltafin exposes an OpenAI-compatible API endpoint. Any client capable of calling the ChatGPT API—such as Cursor, Continue, Claude Code, or local scripts—can point at Deltafin as a local Kimi K3 backend.
Measured Performance
Reference medians:
Approximately 100 tokens take around 24 minutes, while a full Kimi K3 reasoning chain of hundreds to thousands of tokens can stretch from tens of minutes to hours. This is unsuitable for production dialogue but viable for research on routing logic, streaming-inference testing, stress-testing K3 internals, and benchmarking experiments.
Position in the "Compute Trickle-Down" Landscape
Over the past three months, the boundary of "small hardware running large models" has been repeatedly extended:
This trajectory indicates that in 2026 H2, both model scale and on-device inference ceilings are rising in parallel. The ceiling for "what can I run on my machine" has shifted from ~70B dense models to 2.8T MoE.
Deltafin's distinctive stance: it does not care about speed; it cares about *can it run*. As a research project, its value lies in proving that the combination of MoE, quantization, and streaming expert loading is a feasible engineering route on consumer hardware. Every subsequent effort to localize trillion-parameter models now has a baseline showing the problem is solvable.
Practical Recommendations
One-Line Summary
Deltafin is not a killer app for Kimi K3; it is an engineering existence proof that a 2.8T-parameter MoE model can run on a 64GB Mac at 14.6 seconds per token, establishing a baseline for ultra-large-scale MoE localization as unified-memory hardware evolves beyond M1 Max.