English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

PCD: Closing the Training-Generation Gap in Diffusion Language Models

Forum topic · ✨步子哥 · 2026-08-11

Summary

Diffusion language models (dLLMs) such as LLaDA promise parallel decoding with order-of-magnitude speedups over autoregressive generation, but they share a hidden flaw: the pretraining masking scheme does not match inference-time inputs. During training, prompt and continuation tokens are masked randomly and symmetrically, while at inference the model receives a fully clean prompt and only denoises the continuation. A paper by Xiaocheng Lu, Huabin Liu, Song Guo, and Jianguo Li (arXiv:2608.09424) calls this the Pretraining-Generation Mismatch and proposes Prefix-Conditioned Diffusion (PCD). PCD keeps the prompt clean during training by applying causal attention within the prefix, restricting corruption masks to the continuation, and splitting the loss into autoregressive (prompt) and diffusion (continuation) objectives. Crucially, PCD changes only the training objective: inference code, decoding strategies, and architectures stay untouched. Experiments show a 4.2% relative gain on LLaDA2-Mini (continued pretraining) and 14.2% on Qwen-1.7B trained from scratch, with an ablation attributing gains to sample-level prefix conditioning rather than batch-level objective mixing. Limitations include modest absolute gains, validation on only two backbones, and no released code.

PCD: Closing the Training-Generation Gap in Diffusion Language Models

> Paper: Reducing Pretraining-Generation Mismatch in Diffusion Language Models > Authors: Xiaocheng Lu, Huabin Liu, Song Guo, Jianguo Li > arXiv: https://arxiv.org/abs/2608.09424

A Widely Ignored Gap

If you have trained an autoregressive language model (GPT-style), you know the unwritten rule: the input distribution at training time must match the input distribution at inference time.

At inference, the model receives a clean prompt and generates token by token. Training is the same — the model sees a clean left context and predicts the next token. Training and inference share the same "clean prefix" interface.

This alignment feels so natural that we rarely notice it exists. But diffusion language models (dLLMs) break it.

Diffusion LMs: The Price of Parallel Decoding

Autoregressive models need 1000 forward passes for 1000 tokens. Diffusion language models instead mask all tokens randomly, then recover them in parallel over multiple denoising rounds, potentially generating hundreds of tokens in a few dozen steps.

LLaDA, MDLM, and Plaid are representative works. In 2025, Ant Group's LLaDA scaled diffusion LMs to 100B parameters, proving the path is engineering-feasible.

But dLLMs carry an overlooked crack: the masking scheme during pretraining differs from the masking scheme at inference.

Where the Crack Is

In standard dLLM pretraining, masking is globally random:

  • Prompt tokens may be masked
  • Continuation tokens may also be masked
  • Prompt and continuation are treated symmetrically
  • At inference, however, the model gets a fully clean prompt; only the continuation is masked and denoised. The paper calls this the Pretraining-Generation Mismatch:

    | | Pretraining | Inference | |---|---|---| | Prompt state | Randomly masked | Fully clean | | Continuation state | Randomly masked | Fully masked → progressively denoised | | Prompt–continuation relation | Symmetric (equally masked) | Asymmetric (prompt is condition) |

    The consequence: the model never learns "recover the continuation given a clean prompt" during pretraining. It learns a more general but more scattered task — the analogy is a basketball player who only practices shooting from arbitrary positions, then is asked to make free throws in a game.

    PCD: Aligning the Training Interface to Inference

    The proposed fix, PCD (Prefix-Conditioned Diffusion), is simple: treat the prompt as clean during training and apply diffusion only to the continuation.

    PCD modifies three things:

    1. Attention Mask Standard dLLM pretraining uses bidirectional attention everywhere. PCD makes the prompt causal — continuation tokens see all prompt tokens, but prompt tokens only see preceding prompt tokens. This makes prompt representations cleaner, closer to an autoregressive model.

    2. Corruption Mask Standard dLLM masks the whole sequence randomly. PCD masks only the continuation; the prompt stays clean.

    3. Label Construction The loss is split into:

  • Prompt part: autoregressive loss (next-token prediction)
  • Continuation part: diffusion loss (recover from masking)
Together these align the training-time input interface with the inference-time interface.

Key Design: No New Components

PCD requires no autoregressive decoder, no verifier, and no new inference mode. It is a pure training-objective change: inference runs exactly as with a standard dLLM — no code changes, no extra models, no new decoding strategy. Unlike methods such as speculative decoding or Medusa, all changes stay in training, with zero inference cost.

The paper also separates intra-sample prefix conditioning (within one sample: clean prompt, diffused continuation) from inter-sample objective mixing (mixing AR and diffusion samples within a batch). Experiments show the former is the main contributor; the latter is only an optional knob. The alignment signal comes from sample-level interface alignment, not batch-level mixing.

Results

Two backbones were tested:

LLaDA2-Mini: average over 6 benchmarks, baseline → PCD, a 4.2% relative gain (+2.56 points absolute), in a continued-pretraining setup.

Qwen-1.7B: trained from scratch, 14.2% relative gain (+4.86 points absolute).

Consistent gains across backbones suggest PCD is a general improvement to the dLLM training objective, not a model-specific trick.

Why This Matters

Diffusion LMs are among the strongest candidates to challenge the autoregressive paradigm:

1. Speed: parallel decoding generates hundreds of tokens in tens of steps. 2. Global planning: the model sees global sequence structure during generation. 3. Controllability: diffusion naturally supports infilling, editing, and conditional generation.

Yet dLLMs have long been "good but not quite good enough" — close to but below autoregressive models on benchmarks. This paper's diagnosis: part of the gap comes from train-inference mismatch. PCD repairs part of that crack.

Honest Assessment

Gains are moderate. 4.2% and 14.2% are relative; absolutes are 2.56 and 4.86 points, roughly 0.4–0.8 points per benchmark on average. Meaningful but not disruptive.

Only two backbones. LLaDA2-Mini and Qwen-1.7B are relatively small; whether PCD holds at 7B+ scale needs verification.

No open-source code. No GitHub link, which hinders reproduction.

Mixed settings. LLaDA2-Mini uses continued pretraining (low cost); Qwen-1.7B is from scratch. Gains in both settings suggest PCD works across training stages.

Conclusion

The core insight in one sentence:

Diffusion LM training and inference objectives are inconsistent — training masks all tokens randomly, inference masks only the continuation. PCD aligns the training interface to the inference interface and recovers 4–14% of performance without changing inference.

Autoregressive models align these interfaces for free, so the problem goes unnoticed there. In the diffusion paradigm, alignment is no longer a free lunch and must be explicitly designed. For anyone building non-autoregressive models, the question to ask is: is your training task really the same as your inference task?

---

Paper link: https://arxiv.org/abs/2608.09424

Tags

#diffusion-language-models#pcd#pretraining-generation-mismatch#prefix-conditioned-diffusion#llada#training-inference-alignment#masking-strategy#machine-learning-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633328