D-Cut: Fixing Speculative Decoding Degradation at High Concurrency via Draft-Confidence Dynamic Truncation

> Source: AngelSlim docs — D-Cut (Dynamic Cut) > https://angelslim.readthedocs.io/zh-cn/latest/dcut.html > Related papers: DFlash (arXiv:2602.06036) | AngelSlim technical report (arXiv:2602.21233)

TL;DR

D-Cut observes that speculative decoding can become slower than autoregressive decoding at high batch sizes, because the target model wastes ~60% of its compute verifying draft tokens that are destined to be rejected. The fix: score draft positions globally using the prefix product of draft-model confidences, and only verify the most promising tokens — no target-model modification, no extra training — making speculative decoding effective again under high concurrency.

Key points

• The problem: With DFlash-B16 on Qwen3-8B, speedup vs autoregressive decoding collapses as batch size grows: 2.04× at bs=4, 1.28× at bs=8, 0.80× at bs=16, 0.52× at bs=32, and 0.42× at bs=64. The target model must verify bs × D tokens (D = block size = 16) per step, but only ~6.3 are accepted on average.
• Non-linear verification cost: At high bs, full verification is far more expensive than verifying a subset — on Qwen3-8B (TP1), verifying 25% of tokens takes 55.2 ms at bs=64 versus 133.9 ms for full verification (a 2.42× waste factor).
• Core insight: Draft confidence predicts acceptance. Defining the prefix-product score s_i,k = ∏_{t=1}^{k-1} c_i,t (estimated probability that the first k positions are all accepted), this score is monotonically decreasing within a request, so a global top-K across the batch automatically satisfies prefix constraints.
• Global budget allocation beats per-request truncation: With a 128-token verification budget, confidence top-25% accepts 124.6 tokens (97.3% utilization) vs 101.4 (79.2%) for a fixed 4 tokens per request. Globally, top-K accepts 20–25% more tokens than uniform per-request truncation at equal budget, and verifying only half the tokens retains ~95% of full-verification benefits.
• Formulation: The marginal benefit of keeping position k is P(L_i ≥ k), estimated by the confidence prefix product. Throughput is modeled as U_q = Σ_{r=1}^{K_q} s(r) / C_graph(bs, ρ_q) with K_q = ⌈ρ_q · bs · (D-1)⌉, and the bucket with the highest U_q is chosen.
• Cost curves must be measured: Dense models (Qwen3-8B) show near-linear verify cost in token count (2.42× waste at bs=64), while MoE models (Qwen3.5-35B-A3B, TP2) are much flatter (1.13×). D-Cut profiles the cost table at server startup (~30 seconds) and makes data-driven decisions rather than assuming a ratio.
• Engineering: Each step runs: draft → confidence prefix scores → global ranking → select bucket → target verification with keep depth. Verification depth is discretized into 4 CUDA-graph buckets (25%/50%/75%/100% ratios) to avoid graph misses; overlapping shapes mean very few extra graphs are captured.

Results

Qwen3-8B (Dense, TP1) — speedup vs autoregressive:

| Method | bs=4 | bs=8 | bs=16 | bs=32 | bs=64 | Geo Mean | |--------|------|------|-------|-------|-------|----------| | DFlash-B16 | 2.04× | 1.28× | 0.80× | 0.52× | 0.42× | 0.85× | | D-Cut-B16 | 2.42× | 1.93× | 1.41× | 0.99× | 0.81× | 1.39× | | D-Cut-B8 | 2.34× | 2.05× | 1.59× | 1.13× | 0.92× | 1.51× | | EAGLE-3 | 1.70× | 1.64× | 1.37× | 1.01× | 0.84× | 1.26× |

D-Cut extends DFlash from "degrading at bs=16" to "still 0.81× at bs=64", and beats EAGLE-3 at all batch sizes without training a draft model.

Qwen3.5-27B (Dense, TP4): D-Cut-B16 reaches a 1.99× geometric mean (2.83×/2.55×/1.94×/1.59×/1.41× across bs=4–64), fully surpassing MTP (1.93×), which requires modifying the target model architecture.

MoE models: Steady but smaller gains, since cost curves are flatter — e.g., Qwen3.5-35B-A3B: D-Cut-B16 geo-mean 2.83× vs DFlash 2.62×; Qwen3.5-122B-A10B: 1.83× vs 1.64×.

Why it matters

• Speculative decoding benchmarks are usually run at bs=1, but real serving batch sizes are much higher. D-Cut bridges the gap from "works in demos" to "production-ready".
• Zero-cost integration: no target-model changes (unlike MTP), no draft training (unlike EAGLE), one-time 30-second profiling, compatible with mainstream engines like vLLM.
• Generality: although demonstrated on DFlash (block diffusion), the confidence-based truncation logic applies to any speculative decoding framework whose draft model can output confidence.

Limitations

1. Cost profiling is hardware-dependent — re-profiling is needed when switching GPU models. 2. Discrete buckets (4 ratio levels) may lose some precision; finer scheduling could add gains. 3. Gains on MoE are smaller than on Dense — a model property, not a method flaw.

Relation to the AngelSlim ecosystem

D-Cut is part of AngelSlim's speculative decoding toolkit:

• DFlare: improves draft quality (layer-wise fusion, deeper draft models) — raises the numerator.
• D-Cut: reduces verification cost (confidence-based dynamic truncation) — lowers the denominator.
• The two are complementary and should be enabled together for end-to-end optimal throughput.

References

• D-Cut docs: https://angelslim.readthedocs.io/zh-cn/latest/dcut.html
• DFlash paper: arXiv:2602.06036
• AngelSlim technical report: arXiv:2602.21233
• EAGLE-3: https://github.com/vllm-project/vllm/issues/41823