PCD: Closing the Training-Generation Gap in Diffusion Language Models
> Paper: Reducing Pretraining-Generation Mismatch in Diffusion Language Models > Authors: Xiaocheng Lu, Huabin Liu, Song Guo, Jianguo Li > arXiv: https://arxiv.org/abs/2608.09424
A Widely Ignored Gap
If you have trained an autoregressive language model (GPT-style), you know the unwritten rule: the input distribution at training time must match the input distribution at inference time.
At inference, the model receives a clean prompt and generates token by token. Training is the same — the model sees a clean left context and predicts the next token. Training and inference share the same "clean prefix" interface.
This alignment feels so natural that we rarely notice it exists. But diffusion language models (dLLMs) break it.
Diffusion LMs: The Price of Parallel Decoding
Autoregressive models need 1000 forward passes for 1000 tokens. Diffusion language models instead mask all tokens randomly, then recover them in parallel over multiple denoising rounds, potentially generating hundreds of tokens in a few dozen steps.
LLaDA, MDLM, and Plaid are representative works. In 2025, Ant Group's LLaDA scaled diffusion LMs to 100B parameters, proving the path is engineering-feasible.
But dLLMs carry an overlooked crack: the masking scheme during pretraining differs from the masking scheme at inference.
Where the Crack Is
In standard dLLM pretraining, masking is globally random:
- Prompt tokens may be masked
- Continuation tokens may also be masked
- Prompt and continuation are treated symmetrically
- Prompt part: autoregressive loss (next-token prediction)
- Continuation part: diffusion loss (recover from masking)
At inference, however, the model gets a fully clean prompt; only the continuation is masked and denoised. The paper calls this the Pretraining-Generation Mismatch:
| | Pretraining | Inference | |---|---|---| | Prompt state | Randomly masked | Fully clean | | Continuation state | Randomly masked | Fully masked → progressively denoised | | Prompt–continuation relation | Symmetric (equally masked) | Asymmetric (prompt is condition) |
The consequence: the model never learns "recover the continuation given a clean prompt" during pretraining. It learns a more general but more scattered task — the analogy is a basketball player who only practices shooting from arbitrary positions, then is asked to make free throws in a game.
PCD: Aligning the Training Interface to Inference
The proposed fix, PCD (Prefix-Conditioned Diffusion), is simple: treat the prompt as clean during training and apply diffusion only to the continuation.
PCD modifies three things:
1. Attention Mask Standard dLLM pretraining uses bidirectional attention everywhere. PCD makes the prompt causal — continuation tokens see all prompt tokens, but prompt tokens only see preceding prompt tokens. This makes prompt representations cleaner, closer to an autoregressive model.
2. Corruption Mask Standard dLLM masks the whole sequence randomly. PCD masks only the continuation; the prompt stays clean.
3. Label Construction The loss is split into:
Key Design: No New Components
PCD requires no autoregressive decoder, no verifier, and no new inference mode. It is a pure training-objective change: inference runs exactly as with a standard dLLM — no code changes, no extra models, no new decoding strategy. Unlike methods such as speculative decoding or Medusa, all changes stay in training, with zero inference cost.
The paper also separates intra-sample prefix conditioning (within one sample: clean prompt, diffused continuation) from inter-sample objective mixing (mixing AR and diffusion samples within a batch). Experiments show the former is the main contributor; the latter is only an optional knob. The alignment signal comes from sample-level interface alignment, not batch-level mixing.
Results
Two backbones were tested:
LLaDA2-Mini: average over 6 benchmarks, baseline → PCD, a 4.2% relative gain (+2.56 points absolute), in a continued-pretraining setup.
Qwen-1.7B: trained from scratch, 14.2% relative gain (+4.86 points absolute).
Consistent gains across backbones suggest PCD is a general improvement to the dLLM training objective, not a model-specific trick.
Why This Matters
Diffusion LMs are among the strongest candidates to challenge the autoregressive paradigm:
1. Speed: parallel decoding generates hundreds of tokens in tens of steps. 2. Global planning: the model sees global sequence structure during generation. 3. Controllability: diffusion naturally supports infilling, editing, and conditional generation.
Yet dLLMs have long been "good but not quite good enough" — close to but below autoregressive models on benchmarks. This paper's diagnosis: part of the gap comes from train-inference mismatch. PCD repairs part of that crack.
Honest Assessment
Gains are moderate. 4.2% and 14.2% are relative; absolutes are 2.56 and 4.86 points, roughly 0.4–0.8 points per benchmark on average. Meaningful but not disruptive.
Only two backbones. LLaDA2-Mini and Qwen-1.7B are relatively small; whether PCD holds at 7B+ scale needs verification.
No open-source code. No GitHub link, which hinders reproduction.
Mixed settings. LLaDA2-Mini uses continued pretraining (low cost); Qwen-1.7B is from scratch. Gains in both settings suggest PCD works across training stages.
Conclusion
The core insight in one sentence:
Diffusion LM training and inference objectives are inconsistent — training masks all tokens randomly, inference masks only the continuation. PCD aligns the training interface to the inference interface and recovers 4–14% of performance without changing inference.
Autoregressive models align these interfaces for free, so the problem goes unnoticed there. In the diffusion paradigm, alignment is no longer a free lunch and must be explicitly designed. For anyone building non-autoregressive models, the question to ask is: is your training task really the same as your inference task?
---
Paper link: https://arxiv.org/abs/2608.09424