Paper Overview
Field: NLP Authors: Chiatzen Wang, Bill Xu, Haochen Gu, Hongyang Zhang Published: 2026-10-16 arXiv: 2610.23818
Summary
Speculative decoding typically characterizes speedup by the number of draft tokens accepted per verification round. Lossless verification preserves the target model's output distribution—and thus the expected completion length—making this metric a meaningful efficiency proxy. Relaxed (lossy) verification accepts tokens that lossless verification would reject, potentially altering that distribution.
The authors identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, which largely offsets the apparent gains in accepted tokens per round.
Key Findings
- Evaluation covers existing training-free relaxed verification rules on twelve target–drafter pairs, compared against lossless verification on identical prompts, measuring completion length, verifier rounds, wall-clock time, and task accuracy.
- Under the most permissive setting, relaxed verification accepts on average 1.34x more tokens per round, but generates completions that are 1.54x longer.
- For rules with higher acceptance lengths, tokens accepted only under relaxed verification tend to have lower target-model probabilities, contributing to length inflation.
- For rules that reduce verification rounds, task accuracy often drops substantially.
- Further analysis traces some failures to prolonged reasoning exhausting the budget needed to deliver final answers.
Conclusion
Reporting accepted tokens per round alone is unreliable for relaxed verification. To assess true efficiency, completion length and verification rounds must also be reported.