Key points
- The high-batch-size curse of speculative decoding. Block-diffusion drafts like DFlash look great at bs=1, but at bs=16, 32, 64 they become *slower* than autoregressive decoding on Qwen3-8B (0.80x, 0.52x, 0.42x respectively). Roughly 60% of target-model compute is wasted on draft tokens that will be rejected, and verification cost grows non-linearly with batch size.
- Core insight: draft confidence predicts acceptance. For each request
iand positionk, compute the prefix-product scores_{i,k} = ∏_{t=1}^{k-1} c_{i,t}wherec_{i,t}is draft confidence at positiont. This score estimates the probability that the firstkpositions are all accepted. It is monotonically decreasing within a request, so a global top-K selection automatically respects the prefix constraint. - Global top-K beats per-request truncation. Per-request fixed cutoffs waste budget on hard prompts and starve easy ones. On the same verification budget, global confidence top-K accepts 20–25% more tokens than uniform per-request truncation. Verifying only the top 25% retains ~95% of the gain from full verification.
- Optimal truncation ratio via empirical profiling. Throughput is modeled as
U_q = Σ s(r) / C_graph(bs, ρ_q), whereρ_q ∈ {0.25, 0.50, 0.75, 1.00}is the verification ratio andC_graphis measured at startup. Dense models have steep O(N) cost curves (2.42x slowdown at bs=64), so aggressive truncation wins; MoE models have flatter, more memory-bound curves (1.13x at bs=64 on Qwen3.5-35B-A3B), so gains are smaller. - Implementation. Each step: draft → compute prefix scores → global sort → select bucket → run a pre-captured CUDA Graph for the chosen ratio. The four discrete buckets share shapes at most batch sizes, so the extra graph count is negligible.
- Qwen3.5-35B-A3B: D-Cut-B16 2.83x geo mean vs DFlash 2.62x.
- Qwen3.5-122B-A10B: D-Cut-B16 1.83x vs DFlash 1.64x.
- Bridges demo-to-production gap. Speculative decoding benchmarks usually report bs=1 numbers; real serving runs at much higher batch sizes. D-Cut extends the useful batch-size range from "bs=4 OK" to "bs=64 still profitable" without touching the target model or training a draft.
- Zero integration cost. No target-model modifications (unlike MTP's extra head), no draft-model training (unlike EAGLE-3), one-time ~30s cost-curve profiling at boot, and compatible with vLLM.
- Generalizes. The confidence-based truncation is orthogonal to the draft mechanism, so it applies to any speculative-decoding framework that exposes draft token probabilities.
- Cost curves are hardware-specific; redeploying on a new GPU requires re-profiling.
- Four discrete ratio buckets are a coarse approximation; finer-grained scheduling may yield further gains.
- Benefits on MoE models are smaller (a property of MoE compute, not the method).
- DFlare: improves draft quality (layer-wise fusion, deeper draft).
- D-Cut: slashes verification cost (confidence-based dynamic truncation).
- Combined: better drafts × cheaper verification = maximal end-to-end throughput.
- D-Cut docs: https://angelslim.readthedocs.io/zh-cn/latest/dcut.html
- DFlash paper: arXiv:2602.06036
- AngelSlim technical report: arXiv:2602.21233
- EAGLE-3 / vLLM discussion: https://github.com/vllm-project/vllm/issues/41823
Headline results
Qwen3-8B (Dense, TP1)
| Method | bs=4 | bs=8 | bs=16 | bs=32 | bs=64 | Geo Mean | |--------|------|------|-------|-------|-------|----------| | DFlash-B16 | 2.04x | 1.28x | 0.80x | 0.52x | 0.42x | 0.85x | | D-Cut-B16 | 2.42x | 1.93x | 1.41x | 0.99x | 0.81x | 1.39x | | D-Cut-B8 | 2.34x | 2.05x | 1.59x | 1.13x | 0.92x | 1.51x | | EAGLE-3 | 1.70x | 1.64x | 1.37x | 1.01x | 0.84x | 1.26x |
Qwen3.5-27B (Dense, TP4)
| Method | bs=4 | bs=8 | bs=16 | bs=32 | bs=64 | Geo Mean | |--------|------|------|-------|-------|-------|----------| | DFlash-B16 | 2.85x | 2.23x | 1.54x | 1.14x | 0.91x | 1.59x | | D-Cut-B16 | 2.83x | 2.55x | 1.94x | 1.59x | 1.41x | 1.99x | | MTP | 2.45x | 2.30x | 2.01x | 1.66x | 1.44x | 1.93x |