On-Policy Self-Distillation (OPSD): A Model Teaching Itself

Forum topic · QianXun · 2026-08-20

Summary

On-Policy Self-Distillation (OPSD) lets a single language model act as both teacher and student by conditioning the teacher on the ground-truth answer while the student generates trajectories on-policy. This creates an information asymmetry that produces dense token-level supervision without external teachers, reward models, or chain-of-thought annotations. On Qwen3 (1.7B–8B) with math reasoning data, OPSD matches or surpasses GRPO while consuming roughly 1/8 to 1/100 the tokens, because GRPO requires 8 rollouts at 16k tokens versus OPSD's single 1k-token rollout. Critical ablations include pointwise KL clipping to suppress style-token divergence and pairing a direct-answer student with a thinking-mode teacher. Critical analyses reveal OPSD excels when privileged information (PI) is shared (system prompts, persona) but underperforms on sample-specific PI like per-question answers, because the model averages over PI-conditioned teachers rather than learning to generalize. The follow-up USD framework jointly optimizes token weighting and PI intensity via a single dual variable, yielding consistent +2 point gains across model scales.

Key points

Core idea: OPSD uses the same model as teacher and student. The teacher conditions on the ground-truth answer y* plus the student's prefix; the student sees only the prefix. Token-level divergence between the two distributions provides dense supervision aligned with the student's own on-policy distribution.
Loss formulation (JSD variant):

D(p_T \| p_S)(ŷ|x) = (1/|ŷ|) \sum_n D( p_T(\cdot|x, y*, ŷ_{<n}) \| p_S(\cdot|x, ŷ_{<n}) )

L(θ) = E_{(x,y*)\sim S}[ E_{ŷ\sim p_S(\cdot|x)}[ D(p_T \| p_S)(ŷ|x) ] ]

The teacher is a frozen copy of early-training parameters; gradients flow only through the student's logits.

Efficiency vs GRPO: On Qwen3 math benchmarks, OPSD uses ~1/8 to ~1/100 of GRPO's tokens (single 1k rollout vs 8 rollouts × 16k tokens) while matching or exceeding accuracy. GRPO also suffers gradient starvation when batch reward variance collapses to zero.
Critical ablations in the original paper:
*Pointwise KL clipping* prevents high-divergence style tokens (e.g., wait, think) from dominating the loss—essential because OPSD converges in only hundreds of steps.
*Student/teacher mode pairing*: A non-thinking student (TM-off) paired with a thinking teacher (TM-off student + TM-on teacher) maximizes supervision on math tokens and is the main configuration used.
When OPSD fails (Zhu et al., arXiv:2605.11182):
*Sample-specific PI*: For math problems where the privileged information is the per-question answer, OPSD averages over PI-conditioned teachers rather than learning to generalize. Math500/AIME24/25 showed no stable gains on Qwen3-1.7B with OpenThoughts.
*Drifted prefixes*: When a small student's prefix forces the teacher off-track, GPQA-Diamond accuracy for Qwen3-14B dropped from 62.12% to 45.96% (–16.16 points); 40 originally correct answers turned wrong vs only 8 vice versa.
*Biased TopK reverse KL*: Truncating reverse KL to TopK tokens breaks a cancellation that introduces biased gradients; the model becomes verbose, repeats maybe, and accuracy collapses. Fixes: stop-gradient TopK, renormalized TopK, or sampled-token KL in policy-gradient form.
USD unified framework (Yang et al., Tencent, arXiv:2608.08176): Replaces vanilla OPSD's uniform token weighting and full PI exposure with a constrained optimization. A single dual variable λ (marginal price of learning difficulty) jointly controls token selection and PI strength, updated online in O(T) per batch. Avg@12 % across nine benchmarks: vanilla OPSD 56.4 → USD 58.7 (+2.3). Per-scale gains over GRPO: +4.7 (1.7B), +2.0 (4B), +1.5 (8B). Stable hyperparameters: ε=0.3, KL clip 0.05, LoRA r64/α128, AdamW lr 5e-6, 300 steps.
Variants: DASD (entropy-routed teacher influence), TRD (trajectory refinement), ROSD (error-localized KL), Visual-OPSD / D-OPSD for multimodal PI, and a diffusion-LLM version using model-generated suffixes.
Engineering: Sharing parameters between teacher and student saves ~40–60% GPU memory vs external-teacher on-policy distillation.

When to use

Use for: reasoning compression on strong bases (8B+), system-prompt/persona/style alignment where PI is a shared rule, competition math with capable bases.
Avoid for: per-sample privileged information that is absent at test time, weak bases (<8B) that cannot reliably reason, and scenarios where the student has already drifted off-trajectory.

Open questions

Integrating verification signals (correct vs incorrect attempts) into self-distillation without labeled datasets.
Group/ensemble self-distillation.
Curriculum learning when problems exceed the teacher's comprehension even with PI.
Scaling to multi-turn, long-context settings beyond the current 2k–4k token budget.