dSpark: Schedule Distillation Teaches Diffusion LLMs Which Tokens to Commit First

A Crossword Puzzle Analogy

Imagine solving a crossword puzzle. You can roughly guess every blank, but some answers you're sure of while others are ambiguous. If you can only reveal three blanks at a time, which three should you reveal first?

Intuition says: reveal the three you're most confident about. But think deeper—the essence of crosswords is crossing clues. A moderately-confident across word might be the key that unlocks an entire down-word region, instantly making five or six blanks certain. Revealing three unrelated but very certain words might barely help the rest of the puzzle.

This is exactly the dilemma diffusion language models face today. They can generate multiple tokens in parallel, but the seemingly simple question of "which card to flip first" hides a deep efficiency bottleneck.

The Promise and Bottleneck of Diffusion LLMs

Traditional autoregressive models (GPT, Claude) generate one token at a time; an L-token response requires L forward passes. Diffusion LLMs take a different path: starting from a fully masked sequence, each round unmasks (commits) several positions simultaneously, offering theoretical massive parallel speedup. Models like LLaDA and Dream have shown this path approaches autoregressive quality in language modeling, reasoning, and code generation.

But in practice there's a core bottleneck: the unmasking schedule. Which tokens should be committed each round? Commit too many and irreversible errors contaminate later predictions; commit too few and you waste parallelism.

Existing strategies mostly rely on confidence—unmask whichever position has the sharpest predicted distribution first. But is that really correct?

Predictability ≠ Usefulness

dSpark's core insight in one sentence: the easiest-to-predict token is not necessarily the one that should be committed.

Consider two masked positions i and j. Position i's prediction is extremely sharp, but revealing it barely changes the difficulty of predicting elsewhere. Position j may only have moderate confidence, but once revealed, it strongly hints at several surrounding answers. In structured tasks like math reasoning and code generation, this "intermediate decisions unlock downstream dependencies" phenomenon is especially pronounced.

Pure confidence-based schedules waste early commitments on "safe but informationally redundant" tokens, delaying the truly unlock-critical positions. The result: either more rounds needed, or aggressive commitments that introduce irreversible errors.

dSpark's solution: don't ask "which tokens can be predicted now," ask "which revelations make the remaining tokens easier to predict". Treat decoding as context construction—each committed token not only resolves one position but changes the prediction problem for all remaining positions.

Schedule Distillation

Knowing which tokens to commit is one thing; making the model actually do it is another. dSpark uses schedule distillation in two steps:

1. Construct good schedules offline: evaluate candidate commitments' impact on remaining-position uncertainty, prioritizing tokens whose revelation makes multiple positions certain. 2. Distill the policy into the model: use KL-divergence regularization so the model's outputs, under the same number of forward passes, approach the "good schedule" distributions.

The elegance: constructing good schedules requires extra computation, but after distillation, inference needs no such construction—the model has internalized the "which card first" intuition, like a crossword veteran who spots unlock keys instantly rather than deliberating like a novice.

Results: 10x Speedup Is Real

| Benchmark | Backbone | Speedup | Accuracy | |------|------|--------|--------| | GSM8K | LLaDA-8B | 10.12× | 78.9% (base 78.8%) | | GSM8K | Dream-7B | 5.12× | 83.6% (base 82.1%, +1.5) | | MATH | LLaDA-8B | 6.03× | 32.0% (base 31.7%) | | HumanEval | LLaDA-8B | 6.73× | 39.6% (base 38.1%) | | MBPP | Dream-7B | 3.77× | 56.0% (base 57.0%) |

Notable details:

• 10.12× speedup on GSM8K with accuracy up, not down. This isn't trading quality for speed—good scheduling itself makes generation more accurate, since informative early commitments provide more reliable context.
• Comparison with d3LLM: dSpark's AUP (accuracy under parallelism) on GSM8K is 33.3% higher than d3LLM. d3LLM and dParallel have "confidence rates on wrong predictions" of 22.9% and 21.7% respectively, versus only 15.5% for dSpark—91% of dSpark's confident predictions are correct.
• KL regularization acts as a brake: distillation enables aggressive parallel commitment but risks overconfidence on errors; removing the KL weight significantly drops accuracy in ablations.

Why This Matters

dSpark shows that in generative AI, "what to do first" matters as much as "how to do it". Autoregressive models sidestep this question—left-to-right is the only option. But once parallel generation opens, scheduling quality becomes the new bottleneck. This applies beyond diffusion models to any stepwise system with downstream dependencies: chain-of-thought reasoning, tool-call sequences, agent planning.

dSpark's principle: don't greedily chase immediate certainty; value future information gain. Distillation also echoes AlphaGo—Monte Carlo tree search was too expensive, so the intuition was distilled into a policy network.

Honest Limitations

• Experiments cover only math reasoning and code generation, where inter-token dependencies are strong. For open-ended generation (stories, chat), information-gain signals may be less clear.
• The 10.12× speedup depends on a threshold for tokens committed per round; there's no universal optimal threshold, so deployment requires task-specific tuning.
• Distillation is offline, requiring a teacher schedule. If task distribution drifts after deployment, the learned scheduling intuition may degrade—online adaptation remains unexplored.

Conclusion

dSpark gives diffusion LLMs the "which card to flip first" intuition—not via smarter heuristics, but by treating information gain as a first principle and baking it into the model through distillation. The 10× speedup is real; the more exciting direction is that when generation is freed from the left-to-right chain, "what to do first" itself becomes a learnable problem—and good scheduling is not just faster, but more accurate.

---

Paper: Learning When to Commit: Schedule Distillation for Fast Diffusion Language Models Authors: Aidar Myrzakhan, Zhiqiang Shen (VILA Lab, MBZUAI) arXiv: 2610.35804 Code: github.com/VILA-Lab/dSpark