SimSD: Adding a Temporal Filter to Diffusion Language Models for 7.46x Speculative Decoding Speedup

Forum topic · 小凯 · 2026-06-02

Summary

SimSD (Simple Speculative Decoding in Diffusion Language Models) is a training-free method that brings speculative decoding to diffusion language models. Diffusion LLMs generate text in parallel via iterative denoising, but they lack the temporal causality that speculative decoding requires: their bidirectional attention means candidate tokens have no stable prefix context to verify against. SimSD solves this by inserting draft-model predictions as read-only reference tokens and designing a temporal attention mask—reference tokens cannot see each other or the current step, while current tokens can see all references—plus aligned positional encodings. This lets the target model validate all candidates in a single forward pass, just like autoregressive speculative decoding. On SDAR models across GSM8K, MBPP, TriviaQA, and MMLU, SimSD achieves up to 7.46x decoding throughput improvement while slightly improving accuracy (+1.7% average; GSM8K 72.0% to 74.5%, MBPP 68.5% to 70.2%). It is plug-and-play and composable with KV Cache and blockwise decoding.

Why does ChatGPT answer one token at a time? Not by choice—large language models are autoregressive, meaning each token must wait for the previous one to finish, like waiting in line at a bubble tea shop.

Diffusion language models (dLLMs) are challenging this paradigm. Instead of generating left-to-right, they work like a painter: lay down a blurry draft, then refine it over multiple rounds, correcting many positions at once. This gives them inherent parallel generation potential far beyond autoregressive models.

But one key acceleration technique—speculative decoding—doesn't work with diffusion models out of the box.

Speculative Decoding: The Shortcut for Autoregressive Models

Speculative decoding works like taking a multiple-choice exam with a friend who guesses quickly: a small draft model rapidly generates candidate tokens, and the large target model verifies them in one pass, keeping correct guesses and recomputing the rest. Autoregressive models support this naturally because the causal mask guarantees each token only sees preceding content, giving candidates a stable prefix context for verification.

Diffusion models can't do this. Their attention is bidirectional—every token sees all others, including unfilled [MASK] positions. The "effective context" changes at every denoising step, so there's no stable prefix for verification.

In short: autoregressive models have temporal causality; diffusion models don't. Speculative decoding needs temporal causality—so it couldn't be used.

SimSD: A Temporal Filter for Diffusion Models

SimSD's core idea is simple: if diffusion models lack temporal causality, build it artificially. Three steps:

Step 1: Introduce reference tokens. The draft model's candidate tokens are explicitly inserted into the input sequence as "reference context." They are not prediction targets for the current denoising step—they just carry the draft model's prior guesses.

Step 2: Design a temporal attention mask. This is the key:

Reference tokens cannot see each other (independent guesses, no "cheating")
Current-step tokens can see all reference tokens (leveraging the draft model's prior)
Reference tokens cannot see current-step tokens (preserving temporal causality)

It's like fitting the diffusion model with a "temporal filter": reference tokens become read-only history that current tokens can consult without being polluted by future information.

Step 3: Align positional encodings. Reference and current tokens share the same positional encodings so the model understands their positions correctly.

With this mechanism, the target model can compute valid logits for all candidates in a single forward pass—exactly like autoregressive speculative decoding.

Training-Free, Plug-and-Play

A pleasant surprise: SimSD requires no additional training. It only modifies the attention mask design and copies a few hidden states, so it plugs directly into existing dLLM inference pipelines. It also composes with other acceleration techniques—KV Cache, blockwise decoding, all welcome.

It's like swapping in a more efficient gearbox without touching the engine, with no conflicts with other modifications.

7.46x Speedup, Quality Not Just Maintained But Improved

Experiments were run on SDAR-series diffusion language models across four benchmarks: GSM8K (math reasoning), MBPP (code generation), TriviaQA (QA), and MMLU (general knowledge).

Decoding throughput improved up to 7.46x from speculative decoding
Generation quality improved: average accuracy is 1.7% higher than baseline
On GSM8K, accuracy rose from 72.0% to 74.5%; on MBPP, from 68.5% to 70.2%

Why does quality improve? The draft model's candidates aren't always right, but they provide a reasonable prior that helps the target model converge to correct answers faster during denoising—like a friend's guessed exam answers giving you a starting point that's more efficient than starting from zero.

Why This Matters

Diffusion language models are one of the most active directions in AI today. Commercial models like Mercury and Dream have demonstrated the architecture's speed potential. But without multipliers like speculative decoding, that speed advantage gets diluted in long-form generation.

SimSD proves the technical gap between diffusion and autoregressive models isn't insurmountable. With a cleverly designed attention mask, we can keep the parallel advantages of diffusion models while reusing the mature acceleration ecosystem built for autoregressive models.

This "cross-architecture transplant" approach may be more valuable than inventing new methods from scratch. Standing on the shoulders of giants beats rebuilding the ladder.

---

Paper: SimSD: Simple Speculative Decoding in Diffusion Language Models Code: github.com/airevo2/SimSD-release Authors: Junxia Cui, Haotian Ye, Runchu Tian, et al. (UC San Diego, UIUC, Google, UC Santa Barbara)