Learning from the Self-Future: How d-OPSD Enables Self-Evolution in Diffusion Language Models

> Luo, Y., Chen, Z., Wang, H., Hu, X., Zhang, Y., Sha, Z., & Liu, S. *Learning from the Self-future: On-policy Self-distillation for dLLMs.* Tsinghua University, Technical University of Munich, Nanyang Technological University, et al. arXiv:2606.18195, 2026.

1. The Left-to-Right Bias of AR Self-Distillation

On-policy Self-Distillation (OPSD) has proven powerful for autoregressive (AR) LLMs: a single model acts as both student and teacher, with privileged information (e.g., reference answers) injected to provide dense supervision, beating RLVR in sample efficiency.

But existing OPSD methods carry a deep-rooted bias: they are designed for left-to-right generation.

Privileged information is appended to the prompt as prefix conditioning. The teacher sees prompt + reference and generates the next token left to right. The student sees only the prompt and samples the whole sequence. KL divergence is then computed at every token position to pull the student's next-token distribution toward the teacher's.

This logic fits AR models perfectly, since AR can only do prefix conditioning p(suffix|prefix).

Diffusion language models (dLLMs) are fundamentally different: they generate sequences non-autoregressively, predicting all tokens in parallel and progressively revealing them through iterative denoising. This means dLLMs naturally support bidirectional conditioning: not only p(suffix|prefix) but also p(prefix|suffix).

Force-fitting AR-style OPSD onto dLLMs is like driving a right-hand-drive car with left-hand-drive logic — it works, but it is far from optimal.

2. The Two Keys of d-OPSD

The paper's core contribution is redesigning the two central components of OPSD to fit dLLMs' generation process.

Key 1: Suffix Conditioning — Learning from the Self-Future

AR models can only place the reference answer after the prompt as prefix conditioning. The core insight of this paper: inject the model's own fully generated answer as a suffix condition into the teacher's input.

Concretely:

• The student samples a complete T-step trajectory, yielding a final answer y^0
• At each step t, the teacher's input randomly reveals a fraction of tokens from y^0 (keep ratio ρ_teacher = 0.25), replacing the corresponding positions in the current masked state
• The student's input is just the current denoising state y^t, with no privileged information

The teacher can thus "peek" at fragments of the student's final answer at every step, making more accurate predictions based on future information, while the student denoises honestly from the current state.

The paper likens this to human self-reflection — "if I could redo it knowing the ending, what would I do differently?" Crucially, the "future" here is not an external reference answer but the student's own generated answer, preserving the on-policy nature: all data comes from the student's own exploration.

Key ablation (Table 4):

| Method | GSM8K | MATH500 | |:---|:---|:---| | AR-style OPSD (prefix conditioning + reference answer) | 78.4 | 33.4 | | d-OPSD (suffix conditioning + self-generated answer) | 81.0 | 37.2 |

The AR-style scheme adds almost no new knowledge (Overlap Top-K_t ≈ 1), while d-OPSD's teacher provides rich transferable reasoning patterns.

Key 2: Step-level Divergence — Step-Level Supervision for Iterative Denoising

AR OPSD uses token-level KL divergence at each position. This does not fit dLLMs, which at each step predict distributions for all masked positions simultaneously and then reveal only the most confident top-k tokens.

The paper proposes Step-level Divergence:

\[\mathcal{L}_t = \frac{1}{|K_t|} \sum_{i \in K_t} D_{KL}\left(P_{\text{student},t}^i \| P_{\text{teacher},t}^i\right)\]

where K_t is the set of top-k token positions selected for revelation at the current denoising step. The overall objective averages per-step losses over the trajectory:

\[\mathcal{L}_{\text{OPSD}} = \mathbb{E}_x\left[\frac{1}{T}\sum_{t=1}^T \frac{1}{|K_t|}\sum_{i \in K_t} D_{KL}\left(p_\theta(y^i|y_{\text{student},t},x) \| p_\theta(y^i|y_{\text{teacher},t},x)\right)\right]\]

This matches dLLMs' iterative denoising: each step is an independent Markov transition, so supervision should be step-level, not token-level.

A crucial detail: the top-k subset K_t is selected from the teacher's distribution, not the student's (Table 7). This forces the student to align with the teacher's most confident decisions, providing a stronger learning signal.

3. Results: ~10% of the Steps, Better Performance

Experiments are run on LLaDA-8B-Instruct across four reasoning tasks: GSM8K, MATH500, Countdown, and Sudoku.

Performance

| Method | GSM8K (256) | MATH500 (256) | Countdown (128) | Sudoku (128) | |:---|:---|:---|:---|:---| | Base Model | 76.0 | 31.8 | 20.3 | 11.5 | | diffu-GRPO (RLVR) | 79.8 | 37.2 | 33.2 | 18.4 | | d3LLM (off-policy self-distillation) | 72.7 | 30.6 | 36.7 | 9.1 | | d-OPSD | 81.0 | 37.2 | 37.9 | 23.9 |

d-OPSD outperforms RLVR and SFT baselines across the board, achieving SOTA on most tasks.

Sample Efficiency (the highlight)

| Method | GSM8K | MATH500 | Countdown | Sudoku | |:---|:---|:---|:---|:---| | diffu-GRPO steps | 7,700 | 6,600 | 5,000 | 3,800 | | d-OPSD steps | 425 | 100 | 175 | 425 | | Efficiency gain | 18.1× | 66× | 28.6× | 8.9× |

d-OPSD needs only about 10% of RLVR's optimization steps to reach better performance, because dense teacher supervision makes each gradient step more informative.

4. Fundamental Differences from d3LLM / Cd4lm

| Dimension | d3LLM / Cd4lm | d-OPSD | |:---|:---|:---| | Source of privileged information | Ground truth from static datasets | Student's self-generated answers | | On-policy nature | ❌ Off-policy | ✅ On-policy | | Trajectory authenticity | Single-step fabricated "fake trajectories" | Real T-step denoising trajectories | | Supervision density | Sparse (single-pass supervision) | Dense (supervision at every step) | | Essence | An SFT variant | True on-policy distillation |

d3LLM scores only 72.7 on GSM8K versus d-OPSD's 81.0, showing that on-policy + full-trajectory supervision design choices matter.

5. Failure Mode: Policy Collapse

Like RLVR, d-OPSD suffers from policy collapse: after peak performance, training performance degrades catastrophically.

The paper hypothesizes the cause is the mode-seeking property of reverse KL — the student tends to match a narrow mode of the teacher distribution, losing diversity once over-narrowed and stalling learning.

Mitigations:

• Pointwise KL clipping (threshold 0.05): caps individual token KL contributions to prevent early collapse
• Fix teacher strategy: freezes the teacher at the initial policy to prevent degradation alongside the student
• Correctness filtering: computes loss only on samples with correct answers to reduce noise

These stabilize training, but collapse can still occur after the peak — a shared challenge for OPSD and RLVR requiring more fundamental solutions.

6. Core Insight

The paper's deepest lesson is not a specific trick but respect for the intrinsic nature of dLLMs.

AR and dLLMs are not "the same thing with different implementations" — their generation mechanisms are fundamentally different:

• AR: left-to-right, prefix conditioning, token-level prediction
• dLLMs: bidirectional, arbitrary order, step-level prediction

d-OPSD's success comes from rethinking every design choice:

• Prefix conditioning → suffix conditioning (leveraging bidirectional modeling)
• Static reference answers → self-generated answers (truly on-policy)
• Token-level supervision → step-level supervision (matching iterative denoising)

---

> Reference > Luo, Y., Chen, Z., Wang, H., Hu, X., Zhang, Y., Sha, Z., & Liu, S. (2026). Learning from the Self-future: On-policy Self-distillation for dLLMs. arXiv:2606.18195.