The Giant Bound by Chains: The Iron Law of Autoregressive Models
There is an almost ubiquitous metaphor for how language models generate text: they work like a typewriter. ChatGPT always starts from the first character and types one character after another; each next character must wait for the previous one. This strictly sequential, word-by-word generation is known in AI as the autoregressive (AR) paradigm.
Since the Transformer architecture emerged in 2017, the AR paradigm has dominated NLP. GPT, LLaMA, Claude, Gemini—every household name faithfully follows it. Their success is evident: they write poetry, code, solve math, and discuss philosophy and law.
Yet this typewriter has a deep-rooted limitation: it can only look forward, never back.
Imagine writing an article and realizing at character 50 that character 3 was wrong—you can only keep going forward. AR models face the same dilemma: every decision is hostage to what came before, lacking the ability to "review globally and sculpt as a whole." Ask one to write a sentence backwards and it struggles; fill-in-the-blank with both left and right context also underperforms.
Just as AR models were marching triumphantly, a wave from the "image generation" world is quietly rolling toward the shores of language.
Magic in the Darkroom: The Past and Present of Diffusion Models
If you have used Midjourney, DALL-E, or Stable Diffusion, you have seen this: a blurry, noise-filled image gradually sharpens into a lifelike picture within seconds. That is how diffusion models work.
Their inspiration comes from physical diffusion—like a drop of ink spreading in water. The "forward process" adds random noise to an image until it becomes pure, information-free static. The reverse process is the real magic: a neural network is trained to denoise step by step, recovering or creating a brand-new image—like a darkroom technician developing an overexposed, blurry negative into a crisp photograph.
Diffusion models have triumphed in images. But there is a catch: images are continuous pixel values, while language is discrete symbols—a pixel can be 0.5, between black and white, but there is no "intermediate state" between the words "apple" and "banana."
This gap is the biggest obstacle for diffusion models entering language. Crossing it required entirely new ideas.
> Glossary: Diffusion Model > > A diffusion model is a generative AI model whose core idea is "destroy first, rebuild later." During training, it learns how data (images, audio, etc.) is gradually noised into pure random signals; at inference, it starts from pure noise and denoises step by step to generate new data. It is the dominant technology in image generation today, powering Midjourney and Stable Diffusion.
Bridging Two Worlds: Continuous vs. Discrete
Researchers have taken two very different paths around the "language is discrete" problem.
Path 1: Force a bridge through continuous space. Translate text into continuous embedding vectors, run diffusion in vector space, then translate back. This is the continuous-space diffusion language model approach, exemplified by Stanford's Diffusion-LM and later TESS 2. The advantage: it inherits mature image-diffusion techniques. The fatal weakness: the two "translations" inevitably introduce semantic loss—like translating a classical poem to English and back to Chinese, losing subtle flavor.
Path 2: Diffuse directly in the discrete word world. Rather than assimilating text, invent a diffusion mechanism designed for words. The key idea: use the [MASK] token to play the role of "noise"—gradually replace words with [MASK], then progressively restore them.
This line is called masked diffusion models (MDM), or discrete diffusion models. It operates in the native token space with no semantic loss, producing more natural text. It is now the absolute mainstream.
Researchers at KAIST proposed an elegant third path—Riemannian Diffusion Language Models (RDLM). Using Riemannian geometry, they map discrete words into a curved continuous mathematical space, enabling continuous diffusion while preserving semantic integrity and eliminating harsh "jumps" between words.
| Route | Representative models | Core idea | Strengths and limits | |---------|---------|---------|-----------| | Continuous-space DLMs | Diffusion-LM, TESS 2 | Map tokens to a continuous embedding space for noise-denoise, then map back | Inherits mature image-diffusion tech, but suffers semantic mapping loss | | Discrete-space DLMs | D3PM, LLaDA, Dream | Operate directly on tokens via transition matrices that replace tokens with [MASK], then recover | No semantic loss; now the mainstream | | Riemannian DLMs | RDLM | Map discrete words to a curved continuous space via Riemannian geometry | Smooth diffusion paths; elegant theory |
> Glossary: [MASK] Token > > [MASK] is the special placeholder used by BERT-style models for cloze tasks. In masked diffusion models it gains a new meaning: "a position whose content is not yet decided." Generation is the process of turning a sequence of all-[MASK] into meaningful text.
The Challenger to the Throne: LLaDA
In 2024, researchers at Renmin University of China's Gaoling School of AI threw a stone that shook the AI world. LLaDA (Large Language Diffusion with mAsking) is the first diffusion large language model capable of genuinely fluent conversation with humans.
The name itself is a pun—"LLaDA" sounds like "llama," the flagship of open-source AR language models—a not-so-subtle declaration of ambition.
LLaDA's core idea in one sentence: replace "next-token continuation" with "bidirectional context unmasking."
Imagine a crossword master. An AR model fills from left to right, seeing only completed cells. LLaDA covers all cells first, then—referring to the clues (your prompt)—observes all covered cells simultaneously and gradually uncovers them until the grid is complete.
Training: 1. Take a prompt-response sample; 2. Randomly apply [MASK] to the response portion; 3. Train the model to predict what belongs at masked positions given the prompt and partially masked response.
Generation runs in reverse: 1. Start from a fully [MASK]-covered response; 2. Run multiple denoising rounds, each inferring some positions from the prompt and known content; 3. After several steps, all [MASK] tokens are replaced and the full answer emerges.
It is like a sculptor facing a whole block of marble: an AR sculptor can only chisel left to right, while LLaDA's sculptor examines the entire stone and carves holistically, so all parts of the work are naturally coordinated.
LLaDA's performance turned heads. At 8B parameters it matches LLaMA3-8B on many mainstream benchmarks—a newcomer nearly catching a heavily engineered incumbent with a vast ecosystem is already remarkable. Better still, on GSM8K (grade-school math reasoning) LLaDA leads LLaMA3-8B by a full 5%, and on "reversed language tasks" (writing sentences backwards, reverse alphabetical order) it even surpasses OpenAI's GPT-4o.
> Glossary: GSM8K Benchmark > > GSM8K (Grade School Math 8K) is a benchmark of 8,000 grade-school math word problems, widely used to evaluate mathematical reasoning and multi-step computation. High scores imply a relatively reliable chain of logical reasoning.
A Galaxy of Stars: The DLM Family
LLaDA is not alone. Between 2024 and 2025 a batch of notable works has emerged.
Dream-7B, from the University of Hong Kong, applies deeper engineering optimization and improved training strategies atop LLaDA, surpassing both LLaDA-8B and LLaMA3-8B across multiple mainstream benchmarks. The race between AR and diffusion is intensifying.
Industry entry adds fuel: Inception Labs' Mercury and Google DeepMind's Gemini Diffusion are the industrial closed-source representatives, achieving astonishing inference speeds of thousands of tokens per second, with performance comparable to GPT-4-class systems. When tech giants start betting on diffusion, the outcome of the paradigm war may be sealed.
Another notable innovation is LLaDA-MoE. In September 2025, the team added a Sparse Mixture of Experts (MoE) architecture: activating only 1.4B parameters achieves the performance level of the 3B dense AR model Qwen2.5-3B. This suggests a double efficiency dividend: *diffusion's parallelism × MoE's sparse computation*.
> Glossary: MoE (Mixture of Experts) > > MoE is an architecture composed of multiple "expert" subnetworks. For each input, a router activates only a few most relevant experts rather than all parameters—like specialists joining a project on demand instead of everyone attending every meeting. It preserves capability while greatly cutting per-inference compute cost.
| Model | Institution | Size | Architectural highlight | Signature achievement | |---------|---------|---------|---------|-----------| | LLaDA-8B | Renmin University of China | 8B | Masked diffusion, bidirectional context | Conversable DLM pioneer; GSM8K +5% over LLaMA3-8B | | LLaDA-MoE | Renmin University of China | Sparse MoE | Diffusion + sparse MoE | 1.4B active params reach Qwen2.5-3B level | | Dream-7B | University of Hong Kong | 7B | Optimized training | Surpasses LLaDA-8B and LLaMA3-8B on multiple benchmarks | | Mercury | Inception Labs | Undisclosed | Industrial closed-source | Thousands of tokens/sec, GPT-4-class performance | | Gemini Diffusion | Google DeepMind | Undisclosed | Industrial closed-source | Google's first large diffusion LM | | RDLM | KAIST | — | Riemannian diffusion | Smooth diffusion over discrete words |
The Awakening of Parallelism: Speed Revolution and Inference Acceleration
If performance is a DLM's looks, speed is its sharpest weapon.
AR models face an almost insurmountable physical bottleneck: strictly serial, token-by-token generation. Generating 1,000 words requires 1,000 forward passes, each waiting on the last—a single-lane highway where even powerful cars are limited by road structure.
DLMs natively support parallel generation: each denoising round predicts words at all masked positions simultaneously. It is like expanding a single lane into a maze of overpasses.
Combined with dedicated acceleration techniques, results are striking:
- Step distillation: a "student model" mimics the combined effect of many denoising steps, compressing dozens of steps into a few or even one.
- FreeCache: researchers creatively ported KV cache techniques—originally designed for AR models—to the semi-autoregressive decoding of DLMs, caching and reusing computed intermediate states for up to 34x inference speedup.
- Semi-autoregressive compromise: exemplified by SSD-LM, it balances pure serial AR and fully parallel diffusion—splitting text into chunks that are parallel within and serial across, trading off flexibility and parallel efficiency.
- Parallel planning: draft multiple action plans in one inference pass, then optimize globally.
- Bidirectional consistency: full-context generation yields replies more consistent with the whole conversation.
- Real-time capability: industrial models like Mercury, at thousands of tokens/sec, fully meet real-time dialogue needs.
> Glossary: KV Cache > > The Key-Value cache is the standard AR inference acceleration: when generating each new token, Transformers need Key/Value vectors for all prior tokens; the cache stores them to avoid recomputation. Porting it to diffusion requires handling the fact that sequence content changes during denoising—a non-trivial engineering innovation.
Can Diffusion "Think"? The Evolution of Reasoning
A long-criticized weakness of DLMs was poor compatibility with Chain-of-Thought (CoT) reasoning—where models "think aloud" step by step before answering. Diffusion prefers parallel generation, while reasoning chains require earlier steps to precede later ones—an inherent tension.
Researchers did not give up:
Diffusion-of-Thought (DoT) treats each "thought" as an intermediate noisy state of diffusion—thoughts are inferred in parallel, mutually constraining during denoising, converging into a logically self-consistent chain. Like several detectives investigating different leads simultaneously, then piecing together the truth at a round table.
A more ambitious breakthrough comes from reinforcement learning. Algorithms like Diffu-GRPO and VRPO port policy-gradient and preference optimization (RLHF-style) frameworks to DLMs. Results are encouraging: RL-finetuned DLMs improve markedly on math reasoning and code generation.
> Glossary: RLHF and Preference Optimization > > RLHF (Reinforcement Learning from Human Feedback) is the key technique making models like ChatGPT helpful and obedient: the model generates candidate answers, humans (or another AI) judge which is better, and the preference signal optimizes the model. Applying it to diffusion requires solving how to score intermediate denoising states—a frontier alignment challenge.
These advances suggest DLMs' "thinking" can be cultivated—and may carry advantages inherently different from AR models: like a right-brained painter of holistic intuition versus a left-brained logician, each with its own strengths.
Multimodal Fusion: Language Meets Vision and Sound
MMaDA (Multimodal Masked Diffusion with Adaptive masking) is an ambitious unified multimodal diffusion model handling three very different tasks in one framework:
| Task | Baseline | MMaDA result | |---------|---------|---------| | Text-only reasoning (QA, math) | LLaMA3 series | Surpasses | | Multimodal understanding (captioning, VQA) | Show-o | Surpasses | | Text-to-image generation | SDXL | Surpasses |
Three capabilities in one model, each state-of-the-art at its scale—previously requiring three separate specialized models.
LLaDA-V extends LLaDA to vision-language tasks, giving the model "eyes" for mixed image-and-text input.
ThinkDiff proposes a novel combination: the "reasoning brain" of vision-language models (VLM) with the "creative hand" of diffusion—performing visual logical reasoning, then creating images from the results, e.g., solving a visual logic puzzle and then drawing the answer.
In audio, DIFFA (Diffusion-based large Audio language model)—the first diffusion-based large audio language model—surpasses multiple AR baselines on spoken-language understanding benchmarks. Sound, text, and image are converging under the diffusion banner.
Into Reality: Applications
Code Generation: The Art of Fill-in-the-Blank
DiffuCoder leverages bidirectional context for "code infilling"—given a function's beginning and end, it fills the middle, attacking from both ends rather than one. It achieves human-evaluation-comparable performance among open-source models.
Summarization: Important Information First
InfoDiffusion prioritizes high-information words in early denoising steps, matching how humans write "main idea first, details later."
Agents and Real-Time Interaction
DLMs have natural advantages for AI agents:
A Long Road: Deep Challenges Remaining
Challenge 1: Length flexibility. AR models naturally emit an EOS token when done. But non-autoregressive parallel generation requires fixing sequence length in advance—like a crossword, you must know the grid size first. Too short truncates content; too long produces filler. Semi-autoregressive methods are the mainstream compromise, but stopping "naturally like AR" while enjoying parallel efficiency remains an open problem.
Challenge 2: Quantization for deployment. Production models are quantized from float to low-bit integers to cut memory and latency. DLM denoising produces many activation outliers—a few extremely large intermediate values—making low-bit quantization harder; quantized DLMs lose more accuracy than AR models, limiting edge/mobile deployment.
Challenge 3: Infrastructure debt. A decade of software stacks (PyTorch, TensorFlow), hardware (GPU tensor-core optimization), and deployment tooling (TensorRT, vLLM) has been built around AR models. DLMs lack dedicated inference engines and hardware support—like gasoline stations unable to serve electric cars. Even with theoretical efficiency advantages, DLMs are dragged down by immature toolchains versus highly optimized AR inference frameworks.
The Future Is Now: Where the Paradigm War Is Heading
Every true paradigm shift—vacuum tubes to transistors, HDDs to SSDs, CNNs to Transformers—was met with doubt, and every time history delivered a surprising answer.
LLaDA, Dream, Mercury, Gemini Diffusion… these names show diffusion is no longer a lab toy but a combat-ready rival to AR models. Key directions ahead:
1. Continued scaling validation (Scaling Law) — whether DLMs keep improving with scale, and whether their parameter-performance curve might be steeper than AR's, will decide the contest.
2. Maturation of unified multimodal frameworks — MMaDA shows diffusion naturally unifies text, image, and audio under one denoising framework, making it a candidate foundation for truly unified multimodal models.
3. Deep fusion with reinforcement learning — as alignment grows in importance, efficient diffusion-specific alignment algorithms are highly valuable research.
4. Dedicated hardware and systems optimization — native hardware support for diffusion inference would rapidly erase today's toolchain disadvantages.
> A final prophecy > > The competition likely ends not with one paradigm annihilating the other, but with division of labor: AR models, with mature toolchains and excellent streaming output, keep dominating real-time chat; diffusion models, with parallelism and bidirectional context, shine in code completion, document generation, creative writing, and multi-agent systems that need "global planning." > > Either way, the world of language models is more exciting than ever.
Epilogue: From Noise to Thought, from Chaos to Order
In 1944, physicist Schrödinger wrote in *What Is Life?*: life feeds on negative entropy—in a universe trending toward thermodynamic disorder, life draws order from its environment to resist entropy and maintain structure.
Diffusion language models are a digital metaphor of this philosophy: starting from chaos fully covered by [MASK], drawing on the "light of order" provided by the prompt, step by step transforming chaos into meaningful language, carving noise into thought.
AR models are travelers moving forward in the flow of time, every step defined by the past, unable to look back or see the whole. Diffusion language models are more like meditating poets—they first gaze at the entire world, feel the vibration of all possibilities, then, from chaos, slowly and holistically awaken a poem from nothingness.
When we discuss AI's future, we are not just discussing parameter counts, benchmark scores, and inference speed. We are asking a deeper question: should machines think more like flowing water, or more like sculpture?
And diffusion language models are answering in a way never seen before.
References
1. Nie, S., Zhu, F., Du, C., et al. (2025). *LLaDA: Large Language Diffusion with Masking.* arXiv preprint arXiv:2502.09992. Renmin University of China — foundational masked-diffusion work, first conversable diffusion LLM. 2. Du, Y., Li, S., Torralba, A., et al. (2023). *Diffusion-of-Thought: Chain-of-Thought Reasoning in Diffusion Language Models.* arXiv preprint arXiv:2402.07754. 3. Zhao, Z., Li, M., Huang, Z., et al. (2025). *MMaDA: Multimodal Large Diffusion Language Models.* arXiv preprint. 4. Ye, J., Guo, Z., Li, Y., et al. (2025). *Dream: Efficient Diffusion Language Model with Reasoning Enhancement.* arXiv preprint. University of Hong Kong. 5. Austin, J., Johnson, D. D., Ho, J., et al. (2021). *Structured Denoising Diffusion Models in Discrete State-Spaces (D3PM).* NeurIPS 2021.