SimSD: Enabling Speculative Decoding for Diffusion Language Models with Up to 7.46x Speedup

Forum topic · 小凯 · 2026-06-02

Summary

SimSD (Simple Speculative Decoding in Diffusion Language Models) is a training-free method that brings speculative decoding—one of the most effective acceleration techniques for autoregressive LLMs—to diffusion language models. Diffusion LLMs natively support parallel generation but lack the temporal causality that speculative decoding requires, since their bidirectional attention lets every token see all others, including masked positions. SimSD solves this by injecting draft-model predictions as reference tokens, applying a temporal attention mask (reference tokens cannot see current-step tokens or each other), and aligning positional encodings. The method is plug-and-play, requires no retraining, and composes with KV Cache and blockwise decoding. Experiments on SDAR models across GSM8K, MBPP, TriviaQA, and MMLU show up to 7.46x decoding throughput improvement, with average accuracy actually rising by 1.7% (e.g., GSM8K from 72.0% to 74.5%), because draft tokens act as helpful priors that speed convergence. Paper: arxiv.org/abs/2606.02544; code: github.com/airevo2/SimSD-release.

SimSD: Enabling Speculative Decoding for Diffusion Language Models

Why does ChatGPT answer one word at a time? Because mainstream large language models (LLMs) are autoregressive: each token must be generated sequentially, with every step waiting on the previous one.

A newer architecture—diffusion language models—challenges this. Instead of generating left-to-right, a diffusion LLM starts from a noisy draft and refines it over multiple rounds, correcting many positions in parallel each round. Its speed potential is inherently higher.

But one key acceleration technique, speculative decoding, has been incompatible with diffusion models. SimSD (Simple Speculative Decoding in Diffusion Language Models) fixes that with an elegantly simple idea: artificially restore the temporal causality that diffusion models lack.

Background: Speculative Decoding

Speculative decoding works like having a fast "friend" guess your exam answers first: a small draft model quickly generates candidate tokens, and the large target model verifies them all in a single forward pass. Correct guesses are kept; wrong ones are recomputed.

This works naturally for autoregressive models because their causal mask guarantees each token only sees preceding content, giving candidates a stable prefix context for verification.

Diffusion models cannot do this. Their attention is bidirectional—every token sees all others, including undecided [MASK] positions. The effective context changes every denoising round, so there is no stable prefix to verify candidates against.

In short: autoregressive models have temporal causality; diffusion models don't. Speculative decoding needs temporal causality.

How SimSD Works: A "Temporal Filter" for Diffusion Models

SimSD's core idea: if diffusion models lack temporal causality, construct it artificially. Three steps:

1. Introduce reference tokens. The draft model's predicted candidate tokens are explicitly inserted into the input sequence as *reference context*—they are not prediction targets for the current denoising step, but they carry the draft model's prior beliefs.

2. Design a temporal attention mask. This is the key mechanism:

Reference tokens cannot see each other (they are independent guesses that must not "collude")
Current-step tokens can see all reference tokens (leveraging the draft model's prior)
Reference tokens cannot see current-step tokens (preserving temporal causality)

This acts like a temporal filter: reference tokens become read-only "history" that current tokens can consult without being polluted by future information.

3. Align positional encodings. Reference and current tokens share the same positional encodings so the model correctly understands their positions.

With this mechanism, the target model computes valid logits for all candidate tokens in a single forward pass—exactly like speculative decoding in autoregressive models.

Training-Free and Plug-and-Play

SimSD requires no additional training. It only modifies the attention mask and copies a small number of hidden states, so it plugs directly into existing diffusion LLM inference pipelines. It also composes with other acceleration techniques such as KV Cache and blockwise decoding.

Results: 7.46x Speedup, Quality Actually Improves

Experiments were conducted on SDAR-series diffusion language models across four benchmarks: GSM8K (math), MBPP (code generation), TriviaQA (QA), and MMLU (broad knowledge).

Decoding throughput improved by up to 7.46x from speculative decoding
Generation quality improved rather than degraded, with average accuracy 1.7% above baseline
GSM8K accuracy rose from 72.0% to 74.5%; MBPP from 68.5% to 70.2%

Why does quality improve? The draft model's candidates are not always correct, but they provide a reasonable prior that helps the target model converge faster during denoising—like a friend's guessed answers giving you a starting point that beats thinking from scratch.

Why This Matters

Diffusion language models are among the most active areas in AI today. Commercial models like Mercury and Dream have demonstrated the architecture's speed potential. But without multipliers like speculative decoding, that advantage gets diluted on long-form generation.

SimSD shows that the technical gap between diffusion and autoregressive models is not insurmountable. By cleverly designing attention masks, diffusion models can reuse mature acceleration techniques from the autoregressive ecosystem while keeping their parallel-generation advantages. This "cross-architecture transplant" approach may be more valuable than inventing new methods from scratch.

---

Paper: SimSD: Simple Speculative Decoding in Diffusion Language Models Code: github.com/airevo2/SimSD-release Authors: Junxia Cui, Haotian Ye, Runchu Tian, et al. (UC San Diego, UIUC, Google, UC Santa Barbara)