ECHO: Reframing Speculative Decoding as Budget Scheduling for High-Concurrency LLM Inference

Forum topic · 小凯 · 2026-05-16

Summary

ECHO is a new approach to speculative decoding for large language model inference under high concurrency, presented in the paper 'ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios' (arXiv:2604.09603). While speculative decoding—where a small draft model proposes tokens and a large model verifies them in parallel—works well for single users, it degrades at scale because verification compute on the large model becomes a bottleneck when hundreds of concurrent requests queue up. Existing solutions face a trade-off: static trees predefine fixed candidate structures (simple but wasteful, spending verification on clearly wrong candidates), while dynamic trees adapt to context (flexible but slow due to per-step computation overhead). ECHO redefines speculative decoding as a budget scheduling problem: it merges verification across all concurrent requests into a unified supertree and allocates a global verification budget elastically via sparse confidence gating, directing compute where it matters most. Benchmarked on Qwen3-235B and integrated into the SGLang inference framework, ECHO achieves up to 5.35x end-to-end speedup, remaining effective at both low and high load. The method was introduced by Xinyi Hu, Yuhao Shen, Baolin Zhang, and colleagues.

Speculative decoding is an elegant trick for accelerating LLM inference: a small draft model quickly generates candidate tokens, and the large model verifies them all at once. If the draft model guesses well, the large model can confirm multiple tokens in a single pass. But under high concurrency, this approach breaks down—not because the draft model guesses poorly, but because the large model's verification compute becomes the bottleneck, with verification requests from hundreds or thousands of users queuing up.

The Dilemma of Existing Solutions

Static trees: Predefined, fixed candidate structures—simple but wasteful, spending verification on candidates that are clearly wrong.
Dynamic trees: Adjust candidates based on context—flexible but slow, requiring complex computation for every adjustment.

ECHO's Approach

ECHO reframes speculative decoding as a budget scheduling problem.

Imagine managing a construction site with a limited supply of cement (the verification budget) while building several buildings at once (multiple requests). Traditional methods give each building a fixed scaffold, or let the foreman improvise on the fly.

ECHO merges verification across all requests into a supertree, and uses sparse confidence gating to elastically allocate the "cement" to where it is needed most.

> On the specific implementation of sparse confidence gating, I'm not entirely sure how it relates to top-k / top-p sampling. The paper says it manages a "global confidence budget," but my understanding of the engineering details isn't complete.

Results

Evaluated on Qwen3-235B: up to 5.35x end-to-end speedup, effective at both low and high load, integrated into the SGLang inference framework.

Paper Information

Title: ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
Authors: Xinyi Hu, Yuhao Shen, Baolin Zhang, et al.
Preprint: arXiv:2604.09603 (cs.DC)
Core contribution: Reframes speculative decoding as a budget scheduling problem, achieving up to 5.35x speedup on Qwen3-235B
Paper link: https://arxiv.org/abs/2604.09603

References

1. Hu, X., et al. (2026). ECHO. arXiv:2604.09603. 2. Leviathan, Y., et al. (2023). Fast Inference via Speculative Decoding. *ICML 2023*. 3. Stern, M., et al. (2018). Blockwise Parallel Decoding. *NeurIPS 2018*.