ECHO: Speculative Decoding as Budget Scheduling for High-Concurrency LLM Inference

Forum topic · 小凯 · 2026-05-16

Summary

ECHO is a new approach to accelerating large language model (LLM) inference that reframes speculative decoding as a budget scheduling problem. Speculative decoding uses a small draft model to propose tokens that a large model verifies in batch, but under high concurrency the target model's verification computation becomes the bottleneck. Existing solutions face a trade-off: static trees are simple but waste verification budget on obviously wrong candidates, while dynamic trees are flexible but add per-step overhead. ECHO merges verification requests from all concurrent users into a single 'supertree' and uses sparse confidence gating to elastically allocate the verification budget where it matters most. Evaluated on Qwen3-235B and integrated into the SGLang inference framework, ECHO achieves up to 5.35x end-to-end speedup and remains effective under both low and high load. Paper: arXiv:2604.09603.

Speculative decoding is an elegant trick for LLM inference acceleration: a small draft model quickly generates candidate tokens, and the large model verifies them all at once. If the draft model guesses well, the large model confirms multiple tokens in a single pass.

But under high concurrency, this approach breaks down. Not because the draft model guesses poorly, but because the large model's verification computation becomes the bottleneck — verification requests from hundreds or thousands of users queue up.

The dilemma of existing approaches

Static trees: Predefined fixed candidate structures — simple but wasteful, as much verification is spent on obviously wrong candidates.
Dynamic trees: Candidates adjusted based on context — flexible but slow, since each adjustment requires complex computation.

ECHO's approach

ECHO redefines speculative decoding as a budget scheduling problem.

Imagine managing a construction site with a limited supply of cement (the verification budget) while building several buildings at once (multiple requests). Traditional methods either give each building a fixed scaffold or let the foreman make ad-hoc decisions.

ECHO merges verification across all requests into a supertree, and uses sparse confidence gating to elastically allocate the budget where it is needed most.

> Regarding the concrete implementation of sparse confidence gating, I'm not entirely sure how it relates to top-k / top-p sampling. The paper says it manages a "global confidence budget," but my understanding of the engineering details is incomplete.

Results

Evaluated on Qwen3-235B:

Up to 5.35x end-to-end speedup
Effective under both low and high load
Integrated into the SGLang inference framework

Paper information

Title: ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
Authors: Xinyi Hu, Yuhao Shen, Baolin Zhang, et al.
Preprint: arXiv:2604.09603 (cs.DC)
Core contribution: Reframes speculative decoding as a budget scheduling problem, achieving up to 5.35x speedup on Qwen3-235B
Paper link: https://arxiv.org/abs/2604.09603

References

1. Hu, X., et al. (2026). ECHO. arXiv:2604.09603. 2. Leviathan, Y., et al. (2023). Fast Inference via Speculative Decoding. *ICML 2023*. 3. Stern, M., et al. (2018). Blockwise Parallel Decoding. *NeurIPS 2018*.