EVICT: Adaptive Draft-Tree Truncation to Make MoE Speculative Decoding Actually Faster

Forum topic · 小凯 · 2026-05-04

Summary

EVICT (Making Every Verified Token Count) is a training-free, hyperparameter-free, lossless method for speculative decoding in Mixture-of-Experts (MoE) large language models. While speculative decoding reliably accelerates dense Transformers, it can paradoxically slow down MoE models like Mixtral: tree-based verification activates different experts per branch, and the union of activated experts grows with tree size, causing verification costs to explode. EVICT addresses this by adaptively pruning the draft tree before target-model verification, keeping only high-value branches whose acceptance probability justifies their expert-activation cost. Low-probability branches are truncated early, reducing the set of experts loaded per verification step while preserving output quality equivalent to standard decoding. The approach requires no retraining or threshold tuning, making it a plug-and-play speedup for MoE inference. The paper is authored by Lehan Pan, Ziyang Tao, Ruoyu Pang, Xiao Wang, Jianjun Zhao, and Yanyong Zhang (arXiv:2605.00342, 2026-04-29). The key insight: in MoE inference, expert activation is a scarce resource, so selectively verifying promising candidates beats brute-force verification of all drafts.

EVICT: Adaptive Verification for MoE Speculative Decoding

Paper: Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding Authors: Lehan Pan, Ziyang Tao, Ruoyu Pang, Xiao Wang, Jianjun Zhao, Yanyong Zhang arXiv: 2605.00342 | 2026-04-29

The Paradox: Speculative Decoding Can Be Slower on MoE Models

For standard Transformer LLMs, speculative decoding works well:

A small draft model quickly generates candidate tokens
The large target model verifies candidates in parallel
Tree-structured verification checks multiple candidate paths at once
Result: faster inference without quality loss

But for MoE models (e.g., Mixtral), tree-based verification runs into a problem:

Different tree branches activate different experts
The union of activated experts grows as branches multiply
Expert activation costs explode
Verification can become slower than standard decoding

In short: more branches → more experts → higher verification cost. The benefits of speculative decoding are canceled out by MoE routing overhead.

EVICT: Truncate the Draft Tree, Keep Only What's Worth Verifying

EVICT's core idea:

> Prune the draft tree before target-model verification, keeping only the high-value branches, so every verified token earns its cost.

Key properties:

1. Training-free — no additional training; applies directly as a plug-and-play module 2. Hyperparameter-free — adaptive, no threshold tuning required 3. Lossless — output is equivalent to standard decoding; it's just faster 4. Adaptive truncation — evaluates each branch's value, cuts low-value branches, reduces expert activation, and lowers verification cost

The intuition: not every candidate deserves verification. Some branches "look wrong" early on — truncating them saves computation without missing correct answers.

Why Adaptive Truncation Solves the MoE Dilemma

The MoE problem:

Expert activation explosion: N branches each activate different experts; the union is large and expensive
Diminishing returns: more branches yield smaller marginal gains, while costs grow linearly

EVICT's fix:

Precision targeting: verify only valuable candidates, avoiding waste on clearly wrong branches
Preserved gains: pruned branches are low-probability ones, so impact on quality is minimal while speedup is substantial
Adaptivity: adjusts to the current state dynamically, with no preset thresholds

Takeaway

As Feynman put it: knowing what *not* to do matters as much as knowing what to do. In MoE inference, expert activation is a scarce resource — allocating it intelligently beats spending it indiscriminately.

Questions worth asking when optimizing LLM inference: 1. Does speculative decoding break down on your MoE model? 2. Is every candidate really worth verifying? 3. Can low-value branches be truncated adaptively? 4. Is compute being allocated smartly?

EVICT reminds us: when compute is limited, deciding what *not* to do matters more than what to do. The fastest systems aren't the ones that verify the most — they're the ones that verify the best.