Learning from the Self-Future: d-OPSD Enables Self-Evolution of Diffusion Language Models

Forum topic · 小凯 · 2026-06-21

Summary

A forum post discusses d-OPSD (on-policy self-distillation for diffusion language models), from Tsinghua University, TUM, NTU and others (arXiv:2606.18195). Existing on-policy self-distillation (OPSD) methods for autoregressive LLMs inject privileged information as prefix conditioning, reflecting a left-to-right bias. d-OPSD redesigns two core components for diffusion LLMs: (1) suffix conditioning, where the teacher receives randomly revealed tokens from the model's own generated final answer, letting it learn from future information, and (2) step-level KL divergence applied to the top-k token positions revealed at each denoising step, selected from the teacher's distribution. On LLaDA-8B-Instruct across GSM8K, MATH500, Countdown and Sudoku, d-OPSD outperforms diffu-GRPO (RLVR) and off-policy baselines like d3LLM, achieving 81.0 on GSM8K and 23.9 on Sudoku. Its key advantage is sample efficiency: it reaches better performance with only about 10% of RLVR's optimization steps (speedups of 8.9x to 66x). The post also covers policy collapse failure modes and mitigations such as KL clipping and teacher freezing.

Learning from the Self-Future: How d-OPSD Enables Diffusion Language Models to Self-Evolve

> Luo, Y., Chen, Z., Wang, H., Hu, X., Zhang, Y., Sha, Z., & Liu, S. *Learning from the Self-future: On-policy Self-distillation for dLLMs.* Tsinghua University, Technical University of Munich, Nanyang Technological University, et al. arXiv:2606.18195, 2026.

1. The Left-to-Right Bias of AR Self-Distillation

On-policy Self-distillation (OPSD) has already proven powerful for autoregressive (AR) LLMs — a single model acts as both student and teacher, with privileged information (e.g., reference answers) injected to provide dense supervision, crushing RLVR in sample efficiency.

But existing OPSD methods carry a deep-rooted bias: they are designed for left-to-right generation.

Privileged information is appended to the end of the prompt as prefix conditioning. The teacher sees prompt + reference and generates the next token from the prefix. The student sees only the prompt and samples the entire sequence itself. KL divergence is then computed at every token position, pulling the student's next-token distribution toward the teacher's.

This logic fits AR models perfectly, since AR can only do prefix conditioning: p(suffix|prefix).

Diffusion language models (dLLMs) are fundamentally different. They generate sequences non-autoregressively — all tokens are predicted simultaneously and progressively revealed through iterative denoising. This means dLLMs naturally support bidirectional conditioning: not only p(suffix|prefix) but also p(prefix|suffix).

Forcing AR-style OPSD onto dLLMs is like applying left-hand-drive logic to a right-hand-drive car — it runs, but it is far from optimal.

2. The Two Keys of d-OPSD

The paper's core contribution is redesigning the two central components of OPSD to genuinely fit dLLMs' generation process.

🔑 Key One: Suffix Conditioning — Learning from the Self-Future

AR models can only place the reference answer after the prompt as a prefix condition. The teacher sees prompt + answer and generates left to right.

dLLMs have no such constraint. The paper's core insight: inject the model's own complete generated answer as a suffix condition into the teacher's input.

Concretely:

The student first samples a full T-step trajectory, yielding the final answer y^0
At each step t, the teacher's input randomly reveals a portion of tokens from y^0 (keep ratio ρ_teacher=0.25), replacing the corresponding positions in the current masked state
The student's input is the current denoising state y^t, with no privileged information

This means the teacher can "peek" at fragments of the student's final answer at every step, making more accurate predictions based on future information, while the student honestly denoises from the current state alone.

The paper likens this to human self-reflection — "if I could do it over knowing the outcome, how would I act?" But the "future" here is not an external reference answer; it is the student's own generated answer, which is truer to the on-policy spirit — all data comes from the student's own exploration.

A key comparison (Table 4) confirms the advantage:

| Method | GSM8K | MATH500 | |:---|:---|:---| | AR-style OPSD (prefix conditioning + reference answer) | 78.4 | 33.4 | | d-OPSD (suffix conditioning + self-generated answer) | 81.0 | 37.2 |

The AR-style scheme brings almost no new knowledge (Overlap Top-K_t ≈ 1), while d-OPSD's teacher provides rich transferable reasoning patterns.

🔑 Key Two: Step-level Divergence — Step-Level Supervision Matching Iterative Denoising

AR OPSD uses token-level KL divergence: at each position i, the difference between student and teacher next-token distributions.

This simply does not apply to dLLMs, because each denoising step does not predict a single token — it predicts distributions for all masked positions simultaneously, then reveals only the most confident top-k.

The paper proposes Step-level Divergence:

\mathcal{L}_t = \frac{1}{|K_t|} \sum_{i \in K_t} D_{KL}\left(P_{\text{student},t}^i \| P_{\text{teacher},t}^i\right)

where K_t is the set of top-k token positions selected for revelation at the current denoising step. The overall objective is the mean per-step loss over the trajectory:

\mathcal{L}_{\text{OPSD}} = \mathbb{E}_x\left[\frac{1}{T}\sum_{t=1}^T \frac{1}{|K_t|}\sum_{i \in K_t} D_{KL}\left(p_\theta(y^i|y_{\text{student},t},x) \| p_\theta(y^i|y_{\text{teacher},t},x)\right)\right]

This design fully matches dLLMs' iterative denoising logic: each step is an independent Markov transition, so supervision should land at the step level, not the token level.

One crucial detail: the top-k subset K_t is selected from the teacher's distribution rather than the student's (Table 7). This forces the student to align with the teacher's most confident decisions, providing a stronger learning signal.

3. Results: ~10% of the Steps, Better Performance

Experiments were run on LLaDA-8B-Instruct across four reasoning tasks: GSM8K, MATH500, Countdown, and Sudoku.

Performance Comparison

| Method | GSM8K (256) | MATH500 (256) | Countdown (128) | Sudoku (128) | |:---|:---|:---|:---|:---| | Base Model | 76.0 | 31.8 | 20.3 | 11.5 | | diffu-GRPO (RLVR) | 79.8 | 37.2 | 33.2 | 18.4 | | d3LLM (off-policy self-distillation) | 72.7 | 30.6 | 36.7 | 9.1 | | d-OPSD | 81.0 | 37.2 | 37.9 | 23.9 |

d-OPSD surpasses RLVR and SFT baselines across the board, achieving SOTA on most tasks.

Sample Efficiency (the Core Highlight)

| Method | GSM8K | MATH500 | Countdown | Sudoku | |:---|:---|:---|:---|:---| | diffu-GRPO steps | 7,700 | 6,600 | 5,000 | 3,800 | | d-OPSD steps | 425 | 100 | 175 | 425 | | Efficiency gain | 18.1× | 66× | 28.6× | 8.9× |

d-OPSD needs only about 10% of RLVR's optimization steps to reach better performance. Dense teacher supervision yields richer learning signals, making each gradient step more effective.

4. Fundamental Differences from d3LLM/Cd4lm

d3LLM and Cd4lm also attempt self-distillation for dLLMs, but d-OPSD differs fundamentally:

| Dimension | d3LLM/Cd4lm | d-OPSD | |:---|:---|:---| | Source of privileged information | Ground-truth from static datasets | Student's self-generated answers | | On-policy nature | ❌ Off-policy | ✅ On-policy | | Trajectory authenticity | Single-step fabricated "fake trajectories" | Genuine T-step denoising trajectories | | Supervision density | Sparse (single supervision) | Dense (supervision at every step) | | Essence | SFT variant | True on-policy distillation |

d3LLM scores only 72.7 on GSM8K, far below d-OPSD's 81.0 — showing that on-policy + full-trajectory supervision design choices are critical.

5. Failure Mode: Policy Collapse

Like RLVR, d-OPSD suffers from policy collapse: after peaking, training performance degrades catastrophically.

The paper hypothesizes the cause is the mode-seeking property of reverse KL — the student tends to match a narrow mode of the teacher's distribution, and after over-narrowing it loses diversity and can no longer learn.

Mitigations:

Pointwise KL clipping (threshold 0.05): caps per-token KL contributions to prevent early collapse
Fix teacher strategy: freezing the teacher at the initial policy to prevent degradation alongside the student
Correctness filtering: computing loss only on samples with correct answers to reduce noise

These stabilize training, but collapse can still occur after the peak. It is a shared challenge for OPSD and RLVR, requiring more fundamental solutions in the future.

6. The Core Insight

The paper's deepest lesson is not any specific trick, but respect for the intrinsic nature of dLLMs.

AR and dLLMs are not "the same thing, just implemented differently." Their generation mechanisms are fundamentally different:

AR: left-to-right, prefix conditioning, token-level prediction
dLLMs: bidirectional, any-order, step-level prediction

Force-fitting AR methods onto dLLMs is like applying bicycle riding logic to a motorcycle — the pedals are there, but the engine works differently.

d-OPSD's success lies in rethinking every design choice:

Prefix conditioning → suffix conditioning (exploiting bidirectional modeling)
Static reference answers → self-generated answers (truly on-policy)
Token-level supervision → step-level supervision (matching iterative denoising)

When a method genuinely fits the model's intrinsic properties, results emerge naturally — no more complex tricks needed, just more correct assumptions.

---

> Reference > Luo, Y., Chen, Z., Wang, H., Hu, X., Zhang, Y., Sha, Z., & Liu, S. (2026). Learning from the Self-future: On-policy Self-distillation for dLLMs. arXiv:2606.18195.