Imagine writing an essay the traditional way: left to right, one character at a time, no going back. If you realize halfway through that an earlier paragraph could be phrased better, too bad—start over.
That is essentially how nearly every large language model works today—autoregressive generation. GPT, Claude, and Qwen all follow this pattern, like an old typewriter: causal attention only lets the model "look forward," never back.
But a fundamental question arises: must language be generated this way?
The Gaoling School of AI at Renmin University and ByteDance's Seed team answered with iLLaDA: a bidirectional, diffusion-based language model trained from scratch that competes head-to-head with autoregressive models across multiple benchmarks.
---
Two Ways to Sculpt
An autoregressive model is like a sculptor carving from foot to head—each cut is permanent, and a mistake in proportion can force a full restart.
A diffusion model is like shaping a lump of clay: rough out a pose first, then repeatedly refine the whole. Every round reviews the entire structure at once—shave where it's too high, add where it's too low. It doesn't write token by token; it guesses and revises all positions together.
This is iLLaDA's core training method: masked diffusion.
Randomly mask words in a sentence and have the model predict them. The mask ratio is sampled uniformly from 0% to 100%—unlike BERT's fixed 15%. Over time, the model learns to reconstruct order from any degree of chaos.
Crucially, iLLaDA uses fully bidirectional attention: every token sees all tokens before and after it. This gives it a natural edge in reverse reasoning and long-horizon planning—like glancing at a math problem's conclusion before deriving the steps.
---
Three "Cheat Codes"
Earlier diffusion LMs (like LLaDA) lagged behind top autoregressive models. iLLaDA closed the gap with three key improvements.
Cheat Code 1: Train Longer, Reuse Data Better
iLLaDA-8B was trained from scratch on 12 trillion tokens—about double LLaDA's data. More importantly, diffusion models are unusually efficient at data reuse: while an autoregressive model may "get it" on a second pass, a diffusion model keeps extracting new information from different masking patterns. During SFT, iLLaDA simply repeated its 25B instruction tokens 12 times, and performance kept improving.
Cheat Code 2: Answer Confident Questions First
Traditional multiple-choice evaluation sums each candidate answer's probability. iLLaDA introduces confidence-based scoring: starting from a fully masked answer, iteratively unmask the single position the model is *most confident* about, then repeat, summing the log-probabilities of those confident choices. This small trick added +0.6 on ARC-Challenge, +2.3 on HellaSwag, and +1.3 on PIQA.
Cheat Code 3: Stop Padding Lengths
Standard SFT concatenates prompts and responses and pads short responses in a batch—wasting compute. iLLaDA instead concatenates all instruction data into one long stream, cuts random 8192-token segments, and masks them uniformly—prompts, responses, and EOS tokens alike. A bonus: generation becomes variable-length, decoding in blocks (e.g., 32 tokens) and stopping when EOS appears.
---
Head-to-Head with Qwen2.5
iLLaDA has 8B parameters vs. Qwen2.5's 7B—roughly fair.
Base models (raw, unfine-tuned):
| Task | iLLaDA-Base vs. Qwen2.5 7B | |------|----------------------------| | MMLU | Better | | BBH | Large lead (+21.6) | | ARC-Challenge | Better (+14.9 vs. LLaDA) | | GSM8K (math) | Better |
Instruct models (after SFT):
| Task | iLLaDA-Instruct vs. Qwen2.5 7B Instruct | |------|------------------------------------------| | GSM8K | Better | | MATH | Close, still behind | | HumanEval (code) | Large lead over LLaDA (+16.5), near Qwen2.5 | | MMLU-Redux | Competitive |
Note: Qwen2.5 additionally underwent reinforcement-learning alignment (RLHF/RLAIF); iLLaDA has not yet. The paper states the remaining gap "is largely because we did not perform RL alignment"—suggesting iLLaDA could match or surpass after adding it.
---
Why This Matters
Autoregressive models are linear, unidirectional, and causal—matching how humans speak, but not how humans think (we juggle multiple threads and revise constantly).
Diffusion models are parallel, bidirectional, and iterative—more like solving a problem on scratch paper: draft, erase, supplement, adjust. This architecture has structural advantages for reverse reasoning, long-horizon planning, and multimodal fusion.
iLLaDA proves one thing: diffusion language models are not a cheap alternative to autoregression, but an independently competitive parallel path.
---
In One Sentence
> If autoregression is a typewriter, diffusion is scratch paper. The former outputs in sequence; the latter allows global revision. iLLaDA has pushed "scratch-paper mode" to a level where it competes on the same stage as "typewriter mode."
---
Paper: https://arxiv.org/abs/2606.25331 Code and weights: https://github.com/ML-GSAI/LLaDA