FreeToken (github.com/FlashML-org/FreeToken, arXiv 2608.16157, Apache-2.0, ~9.1k stars in its first month) is presented as "running 290B+ MoE models on gaming PCs," but the paper's actual contribution is more precise: redesigning the entire serving stack so one personal computer acts as a unified elastic inference platform — model layout, expert residency, CPU-GPU co-execution, agent state reuse, and runtime memory management are all rebuilt around two edge realities: agent workloads keep changing their execution patterns, and no two edge machines have the same resource mix. The author list is the heaviest signal: Song Han (MIT EfficientML), Ion Stoica and Matei Zaharia (the systems minds behind vLLM/Ray/Spark), and Kurt Keutzer — the builders of datacenter inference engines collectively turning to the edge. The paper closes with a line worth quoting verbatim: "FreeToken turns open weights into deployable local software."
Key results: three hardware tiers
| Hardware | VRAM | Model unlocked | Speed | |---|---|---|---| | RTX 4060 laptop | 8GB | Qwen3.6-35B-A3B (NVFP4) | 39.3 tok/s | | RTX 5090 gaming rig | 32GB | DeepSeek-V4-Flash 284B (MXFP4) | 22–25 tok/s | | RTX PRO 6000 workstation | 96GB | GLM-5.2 753B-A40B (NVFP4, 433GB checkpoint) | 2x llama.cpp |
The 4060 figure has a carefully chosen reference point: 39.3 tok/s exceeds hosted Codex's median decode speed of 33 tok/s. Local free inference beats a cloud product on speed for the first time — hence the name FreeToken: tokens are no longer billed; cost collapses into electricity. On RTX 5090, Qwen3.6 reaches 77–83 tok/s, 1.8–2.3x the strongest baselines (llama.cpp, Ollama, KTransformers, MoE-Infinity).
Mechanism anatomy: three codesigns
Prefill: hiding computation inside transfer. The bottleneck for MoE spilling to RAM is PCIe bandwidth. FreeToken uses full-layer-granularity double buffering: while the GPU computes layer *l*, layer *l+1*'s experts stream in over PCIe. An 8192-token prefill block takes 1.19–1.22s — exactly the time to stream a 64.4GB expert pool at 52.7 GB/s (measured PCIe 5.0 ×16 ceiling); expert computation is fully hidden. At 16k tokens, prefill throughput reaches 6.7k tok/s. Disabling double buffering costs 19%/25%/26% at 4k/8k/16k tokens.
Decode: q⋆ = m·B_P/B_H. During decode only a few experts activate per token; most hit an LRU expert cache in VRAM (the paper measured 8 of 12 routed experts hitting). For the *m* missing experts: PCIe-fill the cache, or execute on CPU? FreeToken splits by the ratio of two measured bandwidths: q⋆ = m·B_P/B_H, where B_P is measured host-to-device transfer bandwidth and B_H is measured host memory bandwidth. A 5090 desktop (DDR5, B_P≈B_H) routes ~90% of misses through PCIe cache fills; a 4060 laptop (LPDDR5 at 47.5 GB/s vs PCIe at only 11.8 GB/s) executes ~three-quarters on CPU. One formula, two very different strategies — and the paper stresses the optimal mix "cannot be read off a spec sheet; it must be measured with real tensor shapes on the deployment machine."
Elastic VRAM: the expert cache is a runtime variable, not a load-time constant. No resource on a personal machine is dedicated — browsers and games grab memory anytime. FreeToken resizes the GPU expert cache at scheduler safe points without restarting the engine or reloading weights. Moving from a 5090 server to a dual-channel desktop costs FreeToken only 4% decode speed; llama.cpp loses 20%.
Why LRU wins: routing locality is structural. In an ablation replaying identical routing traces under three placement policies at RTX 5090 capacity (37% of Qwen3.6's expert pool, 11% of DSV4-Flash's), FreeToken's global LRU achieves 16%/39% decode miss rates, KTransformers' static placement achieves 41%/59%, and llama.cpp's routing-blind static split achieves 62%/89%. Adjacent tokens route to overlapping experts — this temporal locality is a gift of MoE model structure, and preserving it yields a 3.9x difference in miss rate.
Semantic anchors: surviving agent edits
Traditional KV caches assume context is an append-only byte stream. But agent harnesses edit context at semantic boundaries: thinking segments are trimmed, tool results are inserted mid-stream, system prompts reordered. The byte-stream assumption collapses on the first edit, forcing full prefix recompute — KTransformers drops 31% throughput between workloads on DSV4-Flash for exactly this reason. FreeToken places semantic-anchor checkpoints at special-token boundaries (thinking start/end, tool-call boundaries): full attention layers reuse KV from before the edit point, recurrent layers (in hybrid architectures) restore from the nearest surviving anchor, and only the truly new suffix is recomputed. Anchor slots are LRU-evicted independently of the KV pool. Context edits become incremental recompute, not full recompute.
Tail TTFT is an availability boundary
The sharpest line in the evaluation: "Tail TTFT is therefore an availability boundary, not a latency statistic." FreeToken's worst TTFT across all load cells stays under 44 seconds; every baseline crosses 150 seconds somewhere — llama.cpp 232s, Ollama 179s, KTransformers 946s — while real agent clients give up at such thresholds (OpenClaw's idle watchdog is 120s; Claude Code's default timeout ~10 minutes). Performance problems ultimately manifest as availability problems.
Positioning and caveats
Quantization (compressing bytes per expert, e.g. 27B dense to 12.9GB) and scheduling (FreeToken, deciding which bytes must be in VRAM when) are orthogonal and stackable — the 433GB GLM-5.2 checkpoint FreeToken runs is itself NVFP4-quantized. Dense models gain nothing from LRU expert caching: no routing, no locality to exploit.
Cold-water footnotes: (1) the threshold moves from VRAM to RAM, it doesn't disappear — 284B needs 32GB VRAM plus ~150GB+ host RAM (the paper's host: 192GB DDR5); 753B needs a 512GB RAM workstation. (2) MoE-only, favoring hybrid-attention architectures; dense flagships don't benefit. (3) 22–25 tok/s decode for 284B is usable for coding agents but won't replace hosted frontier-model speed. (4) Evaluation assumes single-stream interactive use; concurrent serving is another matter. (5) 9.1k stars and 201 open issues in a month; the desktop app ships via flashml.ai while the CLI is open source under rapid iteration.
The closing claim stands on its own: "FreeToken turns open weights into deployable local software" — a machine you already own becomes a frontier-scale inference platform, and the cost of a local token collapses into electricity.
---
*Sources: arXiv 2608.16157 (Yang/Fan/Pan/Xi/Wang/Sun/Keutzer/Han/Zaharia/Xu/Stoica, Berkeley × MIT × Stanford) + github.com/FlashML-org/FreeToken README.*