A human blink takes about 0.3 seconds. There exists a system that, in that time, writes two hundred characters for your AI.
685 tokens per second, dedicated to a single user. That is the official benchmark published by the LightSeek Foundation in mid-October: four NVIDIA GB300 GPUs, running the Qwen3.8-Flash-Next model, under a realistic agent-coding workload. Ordinary users of web-based LLMs typically see token speeds in the tens per second. That's an order-of-magnitude difference — with the same class of models.
The gap hides in the software layer in the middle: the inference engine. Today we take it apart.
The Driver Matters More Than the Car
A large model is the car; the inference engine is the driver. The same model, run by different engines, can differ in speed by several times. This software layer has to answer questions so granular they seem absurd: when hundreds of requests arrive simultaneously, who boards first? When GPU memory is full, who gets evicted? How much KV cache does each request need, and when is it released? Get any of these wrong, and the GPU idles.
The current lineup in this field: TensorRT-LLM is NVIDIA's in-house driver, serving only its own cards; vLLM is the community driver, the most widely deployed; SGLang is another community driver with its own tricks. In May, a newcomer arrived: TokenSpeed, from the LightSeek Foundation. The foundation is a US-registered 501(c)(3) nonprofit, the code is MIT-licensed, and the repository has grown from zero to over 2,600 stars in five months.
The Twelve Stops of a Request
Its most celebrated design is splitting the engine in two. The life-and-death part is written in C++: each request's lifetime is encoded as a finite state machine. The labor part is written in Python: model forward passes and kernel calls, which need fast iteration, stay in the high-level language.
"The first inference engine to separate the control plane from the execution plane" — that's not its own claim; it's how the PyTorch Foundation's official blog described it when bringing it into the ecosystem in August.
Reading the source, a request's life passes through twelve stops: Bootstrapping, Submitted, Prefetching, Prefilling, PrefillAwaitingResult, Decoding, Retracted, Restoring, Finishing, Finished, plus RemoteFill and Ready. Videos say "twelve state machines"; more precisely, it's one state machine with twelve stops, where the prefill and decode stations embed sub-state machines — the count is right, the unit needs correcting.
The implementation is interesting. The twelve stops are encoded in C++'s type system: which station a request is at, and which GPU memory resources it holds, are part of the compile-time type. If you try to write "release the cache to someone else before decoding finishes," the compiler rejects it outright — not a warning during testing, a build failure. The class of incidents that blow up in production at 2 a.m. gets stopped at the moment you write the code.
How 685 Was Assembled
The number needs its context. The test workload is called SWE-Smith, simulating an AI programmer at work: 10–15 turns per conversation, a first turn that ingests 50,000 tokens of context (half a codebase), 800 new tokens per subsequent turn, output capped at 500. This is the typical agent shape — long context, many turns, single-stream output.
On this workload, four GB300s deliver 685.9 tokens per second for one user. The official numbers also go the other way: at 32 concurrent users, each drops to 95.7 tokens/s, but per-GPU throughput per minute rises from 7,300 to 29,000. Both numbers are true — single-user exclusive is the extreme gear; eight people sharing is what you'll actually encounter.
Part of the speed is on the model side. Qwen3.8-Flash-Next is a preview of the Qwen4 architecture: 125 billion total parameters with only 6 billion activated per token, sparse attention that picks only relevant context, and a multi-token prediction module — the engine gambles three tokens ahead each step, the model verifies four at once, and correct guesses are adopted directly. The measured acceptance ratio is 3.23, meaning roughly 8 out of 10 guesses hit. It's a conspiracy between model and engine: the model offers the opportunity, the engine catches it. The engine side has its own optimizations: a compressed and reused index pipeline for sparse attention, kernel-launch/compute overlap, and memory-table offloading — itemized in the blog.
One caveat: all these figures are the official blog's test results. No independent third party has reproduced them yet. Keep that fine print in mind.
The Switzerland of the Engine World
Why would a nonprofit foundation build an inference engine? Look at the table and it's clear. NVIDIA's engine won't optimize for AMD cards; AMD's software stack has no reason to please NVIDIA. When chipmakers build engines, they serve their own hardware first.
TokenSpeed positions itself as "the Switzerland of inference engines": neutral, because it belongs to no chipmaker. The neutrality is concrete — the repo has three kernel directories: generic, AMD, and NPU; both the NVIDIA developer blog and AMD's official blog have written dedicated guides for running models on their own cards; the article on running Kimi K3 on AMD's MI355X and the article on speeding it up on GB300 come from the same team.
Neutrality buys a position called Day 0: the engine runs a model on the day it ships. This year's Day 0 list includes July's Kimi K3, August's 240-billion-parameter Qwen3.8, August's GLM-5.3-Flash, and October's Reflection Beam. Why would model labs hand specs early to a small foundation? Because it's nobody's private army — giving it early access isn't helping a competitor.
The vLLM mega-kernel work covered earlier today and this teardown are two halves of the same elephant: mega-kernels fight idle time inside GPU kernels; TokenSpeed fights idle time in the scheduling layer — both aimed at the agent's "single-stream decoding" scenario. The same Kimi K3 gets mega-kernels upstream in vLLM and two optimized paths (GB300 and MI355X) in TokenSpeed — the race between engines has gotten this close.
To try it yourself: the deployment section of the Qwen3.8 model card already includes the TokenSpeed launch command — no compilation needed, one Docker command and it runs.
---
*Sources: LightSeek official blog "685 tokens/s per user" (2026-10-13, 4×GB300, NVFP4); TokenSpeed repo (lightseekorg/tokenspeed, 2,697★, created 2026-05-06); PyTorch Foundation ecosystem blog (2026-08-26, "the first to separate the control plane from the execution plane"); NVIDIA developer blog and AMD official technical articles (Kimi K3 on MI355X). Performance figures are from the official blog's tests and have not been independently reproduced.*