FreeToken (github.com/FlashML-org/FreeToken, arXiv 2608.16157, Apache-2.0, ~9.1k stars in one month) looks on the surface like "a 290B+ MoE model on a gaming PC." Read closely, the paper does something more precise: it redesigns the entire serving stack around a personal computer as a unified elastic inference platform — model layout, expert residency, CPU-GPU co-execution, agent state reuse, and runtime memory management are all rebuilt around two edge realities: agent workloads keep changing execution patterns, and every edge machine has different resource ratios. The author list is the strongest signal: Song Han (MIT EfficientML), Ion Stoica and Matei Zaharia (vLLM/Ray/Spark), Kurt Keutzer — the builders of datacenter inference engines collectively pivoting to the edge. The paper's closing line: "FreeToken turns open weights into deployable local software."
Key results
| Hardware | VRAM | Model | Speed | |---|---|---|---| | RTX 4060 laptop | 8GB | Qwen3.6-35B-A3B (NVFP4) | 39.3 tok/s | | RTX 5090 desktop | 32GB | DeepSeek-V4-Flash 284B (MXFP4) | 22–25 tok/s | | RTX PRO 6000 workstation | 96GB | GLM-5.2 753B-A40B (NVFP4, 433GB checkpoint) | 2x llama.cpp |
- The 39.3 tok/s on the 4060 laptop exceeds the median decode speed of hosted Codex (33 tok/s) — local free inference beats a cloud product on speed for the first time. Token costs collapse into electricity.
- On RTX 5090, Qwen3.6 reaches 77–83 tok/s, 1.8–2.3x the strongest baseline. Baselines: llama.cpp, Ollama, KTransformers, MoE-Infinity.
- The barrier shifts from VRAM to system RAM, it doesn't vanish: 284B needs 32GB VRAM plus ~150GB-class host RAM (test machine: 192GB DDR5); 753B needs a 512GB RAM workstation.
- MoE-only, favoring hybrid-attention native MoE architectures (DSV4-Flash, GLM-5.2). Dense models gain no routing-locality benefit and are better served by quantization alone.
- 22–25 tok/s for a 284B model is usable for coding agents but slower than API-hosted frontier models for long outputs.
- Evaluation assumes single-stream interactive use; concurrent serving is out of scope.
- The desktop app is distributed via flashml.ai; the CLI is open source but engine internals are iterating quickly (201 open issues).
Mechanism: three codesigns
Prefill — hide computation behind transfer. The PCIe bandwidth bottleneck is addressed with full-layer-granularity double buffering: while the GPU computes layer l, layer l+1's experts stream over PCIe. An 8192-token prefill block completes in 1.19–1.22s — exactly the time to stream a 64.4GB expert pool at 52.7 GB/s (measured PCIe 5.0 x16 ceiling). Prefill throughput hits 6.7k tok/s at 16k tokens. Disabling double buffering drops prefill by 19%/25%/26% at 4k/8k/16k tokens.
Decode — q⋆ = m·B_P/B_H. At decode, most activations hit the on-GPU LRU expert cache (measured: 8 of 12 routed experts hit). Missing experts are split between PCIe fill and in-place CPU execution by the measured ratio of host-to-device transfer bandwidth (B_P) to host memory bandwidth (B_H). The optimal mix differs per machine: a 5090 desktop (DDR5, B_P≈B_H) sends ~90% of misses over PCIe; a 4060 laptop (47.5 GB/s LPDDR5 vs 11.8 GB/s PCIe) executes ~three-quarters on CPU. The paper stresses the ratio must be measured with real tensor shapes on the deployment machine.
Elastic VRAM. Expert caches are rebuilt dynamically at scheduler safe points as other applications claim memory — no engine restart, no weight reload. Moving from a server board to a dual-channel desktop costs FreeToken only 4% decode speed vs 20% for llama.cpp.
Why LRU wins. With identical routing traces at RTX 5090 capacity (37% of Qwen3.6's expert pool, 11% of DSV4-Flash's), FreeToken's global LRU reaches 16%/39% decode miss rates, vs 41%/59% for KTransformers' static placement and 62%/89% for llama.cpp's routing-blind static split — a 3.9x gap. Temporal locality in expert routing is a structural property of MoE models; preserving it in the system determines miss rates.
Semantic anchors: surviving agent context edits
Traditional KV caches assume an append-only byte stream, but agent harnesses edit context at semantic boundaries (trimmed thinking blocks, inserted tool results). That assumption collapses on the first edit, forcing full prefix recomputation — KTransformers loses 31% throughput on DSV4-Flash across agent turns. FreeToken checkpoints semantic anchors at special-token boundaries: full attention layers reuse KV before the edit point, recurrent layers (hybrid architectures) restore from the nearest surviving anchor, and only the new suffix is recomputed. Context editing becomes incremental recomputation, not full recomputation.
Tail TTFT is an availability boundary
The sharpest line in the evaluation: "Tail TTFT is therefore an availability boundary, not a latency statistic." FreeToken's worst-case TTFT stays under 44 seconds across all workload cells; every baseline crosses 150s somewhere (llama.cpp 232s, Ollama 179s, KTransformers 946s) — beyond real agent client timeouts (OpenClaw's idle watchdog: 120s). Performance problems surface as availability failures.
Caveats
Context
Quantization (compressing bytes per expert) and scheduling (deciding which bytes must be in VRAM when) are orthogonal and stackable — FreeToken's 433GB GLM-5.2 checkpoint is itself NVFP4-quantized. When AI agents both produce and consume knowledge locally, inference cost itself begins to collapse.
---
*Source: arXiv 2608.16157 (Yang/Fan/Pan/Xi/Wang/Sun/Keutzer/Han/Zaharia/Xu/Stoica, Berkeley × MIT × Stanford) + the FreeToken GitHub README.*