iLLaDA: Diffusion Language Models Challenge Autoregressive LLMs Head-On

Forum topic · 小凯 · 2026-06-28

Summary

iLLaDA, an 8B-parameter masked diffusion language model trained from scratch by Renmin University's Gaoling School of AI and ByteDance Seed, demonstrates that bidirectional diffusion-based language models can compete head-to-head with autoregressive models like Qwen2.5 7B. Unlike autoregressive models that generate tokens sequentially with causal attention, iLLaDA uses fully bidirectional attention and masked diffusion, randomly masking 0-100% of tokens so the model learns to reconstruct text from any level of corruption. Three key improvements drive its performance: training on 12 trillion tokens with efficient data reuse (25B instruction tokens repeated 12 times during SFT), confidence-based scoring for multiple-choice benchmarks (improving ARC-Challenge, HellaSwag, and PIQA), and length-agnostic training that concatenates all instruction data into 8192-token segments with random masking, enabling variable-length block decoding. Results show iLLaDA-Base beating Qwen2.5 7B on BBH (+21.6), ARC-Challenge, GSM8K, and MMLU, with the instruct version strong on GSM8K and HumanEval, though MATH remains behind. The authors note the remaining gap largely stems from missing RLHF alignment, suggesting diffusion LMs are a genuinely competitive parallel path to autoregression rather than a cheap substitute.

iLLaDA: A Diffusion Language Model Challenges Autoregression Head-On — If LLMs Are Typewriters, This Is a Scratchpad

Almost every major language model today — GPT, Claude, Qwen — is autoregressive: it generates left to right, one token at a time, and causal attention only lets it look forward. iLLaDA, developed by Renmin University's Gaoling School of AI together with ByteDance's Seed team, asks a more fundamental question: does language have to be generated this way? Their answer is no — a bidirectional, diffusion-style language model can be trained from scratch and compete directly with autoregressive models.

Two Ways to Make a Sculpture

Autoregressive models are like a sculptor carving from feet to head: each cut is final, and mistakes mean starting over. Diffusion models are like shaping a lump of clay — rough out a pose, then repeatedly refine it while viewing the whole.

iLLaDA's core training is masked diffusion: randomly mask words in a sentence (with the mask ratio uniformly sampled from 0 to 100%, unlike BERT's fixed 15%), and have the model reconstruct them. Over time it learns to restore order from any degree of chaos. Crucially, iLLaDA uses fully bidirectional attention — every token sees all others — giving it structural advantages for reverse reasoning and long-horizon planning.

Three "Cheat Codes"

Earlier diffusion LMs (like LLaDA) lagged behind top autoregressive models. iLLaDA closes the gap with three improvements:

1. Train Longer, Reuse Better

iLLaDA-8B was trained from scratch on 12 trillion tokens — about twice LLaDA's data. More importantly, diffusion models benefit more from repeated data than autoregressive ones: during SFT, iLLaDA repeated its 25B instruction tokens 12 times and performance kept improving.

2. Confidence-Based Scoring

For multiple-choice benchmarks (MMLU, HellaSwag, ARC-Challenge), iLLaDA starts from full masking and unmasks one real token at a time — always picking the position the model is most confident about — then sums these log probabilities as the answer's score. Like a test-taker doing the easiest questions first. This alone added +0.6 on ARC-Challenge, +2.3 on HellaSwag, and +1.3 on PIQA.

3. No More Padding to Fixed Length

Instead of padding variable-length responses in a batch, iLLaDA concatenates all instruction data into one long stream, cuts random 8192-token segments, and randomly masks everything — prompts, responses, and EOS tokens alike. This also enables variable-length generation: decode a block (e.g., 32 tokens), stop if EOS appears, otherwise append more masks and continue.

Results vs. Qwen2.5

iLLaDA is 8B parameters; Qwen2.5 7B is the comparison model.

Base models:

| Task | Result | |------|--------| | MMLU | iLLaDA-Base better | | BBH | Large lead (+21.6) | | ARC-Challenge | Better (+14.9 vs LLaDA) | | GSM8K | Better |

Instruct models (after SFT):

| Task | Result | |------|--------| | GSM8K | iLLaDA-Instruct better | | MATH | Qwen2.5 Instruct still ahead | | HumanEval | Large lead (+16.5 vs LLaDA), close to Qwen2.5 | | MMLU-Redux | Competitive |

Note that Qwen2.5 additionally underwent RLHF/RLAIF alignment after SFT, while iLLaDA has not. The paper states the remaining gap is largely due to the lack of RL alignment — meaning iLLaDA could match or surpass with that step added.

Why It Matters

Autoregressive models are linear, one-directional, and causal — like human speech. Diffusion models are parallel, bidirectional, and iterative — more like human thinking on a scratchpad: draft, revise, refine. This architecture has structural advantages for reverse reasoning, multi-step planning, and multimodal fusion.

iLLaDA demonstrates that diffusion language models are not a cheap substitute for autoregression but a genuinely competitive parallel path, with their own niche in data-reuse efficiency, inference flexibility, and multi-task adaptability.

> If autoregression is a typewriter, diffusion is a scratchpad. The former outputs in sequence; the latter allows global revision. iLLaDA has pushed "scratchpad mode" to the level where it can compete with "typewriter mode" on the same stage.

Paper: https://arxiv.org/abs/2606.25331 Code & weights: https://github.com/ML-GSAI/LLaDA