TokenRouter: Token-Level Routing Beats 'No Routing' for the First Time

Start with an awkward number. Token-level routing is a compute-saving approach in academia: during decoding, easy tokens go to a small model, hard tokens go to a large model. One such algorithm, Co-LLM, runs at only 3.46 tokens/s in its official implementation; simply skipping routing and sending everything to the large model runs at 134.79 tokens/s. Deployed as-is, the routing algorithm is 39x slower than no routing at all. The official code of another algorithm, R-Stitch, doesn't even run. This is why token-level routing has long stayed in papers: many algorithms, no system to actually run them.

Why it's worth the effort

Each finer level of granularity saves more compute. Production systems today do request-level routing: judge the difficulty of the whole request, pick one model, and commit. But difficulty fluctuates wildly within a single answer—for a math problem, the "derivation" steps need the large model, while "Answer: x=42" is pure copying that a small model handles easily. The R2R algorithm offered striking data: sending just 5% of tokens to a 32B model and the rest to a 1.5B model matches 32B quality; with request-level routing, 40% of requests must be sent wholesale to be equivalent. The gap from 40% to 5% lies entirely in granularity.

Why systems break at token granularity

Inference frameworks like vLLM and SGLang assume "one request belongs to one model end to end": all requests in a batch are locked to the same step-by-step rhythm. Token-level routing means a request can switch models mid-decode, in a data-driven, unpredictable way. Two concrete problems follow: in batch scheduling, when routed requests arrive frequently, the target model is often still processing the previous batch, so new requests queue for admission; in state handoff, nobody has addressed how KV caches and context formats move and recover across models. Fitting routing into speculative decoding frameworks doesn't work—speculative decoding tightly couples two models in a fixed rhythm, while token routing is random access, resulting in idle spinning.

The system

TokenRouter (arXiv 2610.12242), from Tsinghua's NICS lab (Yu Wang's group, corresponding author) and CMU's Yixin Dong—co-first author of SGLang—fills this gap; code released October 8. Three-layer design:

• Programming interface: developers write three functions—route decides which model handles the next token, send packages and transfers request state, receive resumes decoding on the new model. Key idea: "request-side programming, model-side execution"—you describe how a request flows; concurrency, scheduling, and transport are handled by the system.
• Execution layer: one autonomous sub-service per model, with three decoupled asynchronous loops (client communication, decoding, cross-model interaction). A full batch at one model doesn't stall the whole system; cross-model handoffs go through a unified switch-and-resume mechanism.
• Batch scheduling: fragmentation from routing is absorbed by delayed batching; the scheduler's key hyperparameter isn't hand-tuned but derived analytically from a mathematical throughput model.

Results

The clearest picture is on R2R: TokenRouter runs it at 244.56 tokens/s—2.7x the official implementation and, crucially, 1.68x faster than the LLM-only baseline that sends every request to the 8B model. Token-level routing has, for the first time, beaten no routing in a real system. Across four routing algorithms, multiple workloads, and 15 combinations, the paper reports speedups of 2.01x to 64.15x over existing implementations (under the strongest baselines); on four Qwen3 model pairs (0.6B, 1.7B, 4B paired with 8B), it's consistently 2x to 3.2x faster. Layer-by-layer contributions: base implementation at 132.78 tokens/s, extended CUDA graphs to 164, asynchronous execution to 296.86, delayed batching to 372.48. One deployment tip: tensor parallelism on the large-model side is critical for throughput; on the small-model side it doesn't matter.

Takeaway

This is a typical case of an algorithms group building its own systems layer: the same group published the Think-at-hard routing algorithm in 2025, and a year later delivered the serving layer. The code is at thu-nics/TokenRouter, with a vLLM-style interface that can drop in for existing single-model services. To grasp the full significance of token-level routing, remember R2R's number—5% vs 40%. Finer granularity means more compute headroom, but realizing that headroom as speed depends on whether the serving system can catch it.