OPSD: On-Policy Self-Distillation — When an LLM Becomes Its Own Teacher

> Deep research · 2026-08-20 > Main paper: Siyan Zhao et al., *Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models* (UCLA / HKU / Meta Superintelligence Labs, arXiv:2601.18734, 2026-01) > Critical follow-ups: Zhu et al., *The Many Faces of On-Policy Distillation* (UIUC / Renmin Univ. / Peking Univ., arXiv:2605.11182); Yang et al., *Matching Supervision to the Student's Learning Capacity* (Tencent, arXiv:2608.08176)

What it is

OPSD (On-Policy Self-Distillation) lets a single model be its own teacher: by gaining access to the ground-truth answer as extra context, the model supervises — token by token — a version of itself that has *not* seen the answer. No external stronger teacher, no reward model, no human-annotated reasoning chains. It trains on trajectories the model actually generates, not idealized teacher outputs.

The problems it addresses

• SFT: suffers from exposure bias — training and inference distributions diverge, so errors compound autoregressively.
• RL (e.g., GRPO): on-policy and generalizable but expensive (typically 8+ rollouts per prompt), with sparse, sequence-level rewards; all-correct or all-wrong batches give zero gradient.
• Traditional distillation: dense token-level signal but off-policy — the student's inference distribution still drifts from the teacher-generated data.

OPSD aims to combine on-policy distribution realism with dense supervision, without an external teacher.

Mechanism: one model, two roles

1. The student writes first. Given problem \(x\), the student policy \(p_S(·|x)\) autoregressively samples a full solution \(ŷ\) — exactly matching inference-time behavior. 2. Teacher and student share the same prefix. At each position \(n\), both compute the next-token distribution from the same prefix \(ŷ_{<n}\):

• Student: \(p_S(·|x, ŷ_{<n})\)
• Teacher: \(p_T(·|x, y*, ŷ_{<n})\) — the only difference is access to the ground truth \(y*\).
3. Token-wise distribution matching. Minimize the average divergence between teacher and student across all positions (original paper uses generalized Jensen–Shannon divergence \(JSD_\beta\)):
\[D(p_T ‖ p_S)(ŷ|x) = (1/|ŷ|) Σ_n D( p_T(·|x, y*, ŷ_{<n}) ‖ p_S(·|x, ŷ_{<n}) )\]
\[L(θ) = E_{(x,y*)∼S} [ E_{ŷ∼p_S(·|x)} [ D(p_T ‖ p_S)(ŷ|x) ] ]\]

Gradients flow only through the student logits; the teacher is frozen (early-training parameters), stabilizing training and acting as regularization. JSD is preferred over raw KL for symmetry, numerical stability, and a tunable \(\beta\). A cheaper KL variant treats per-position \(\log p_T − \log p_S\) as a fixed advantage inside a policy-gradient objective.

Why it works

The intuition: judging is easier than generating. Comparing your own wrong solution against the reference answer — and spotting where you went astray — is easier than deriving the solution from scratch. The method requires a sufficiently strong base model that can *rationalize* solutions given the answer.

Experiments: real token savings

Setup. Qwen3 base models (including 1.7B for ablations), OpenThoughts math data, competition-level benchmarks (AIME24/25, HMMT, etc.).

| Method | Rollouts per problem | Length | Approx. token cost | |--------|---------------------|--------|--------------------| | GRPO | 8 | 16k | ~128k | | OPSD | 1 | 1,024 | ~1k |

OPSD matches or exceeds GRPO using roughly 1/100 to 1/8 of its tokens; the paper reports 4–8× token efficiency. In GRPO over half of batches had reward standard deviation drop to zero within a hundred steps; OPSD's dense loss sidesteps this.

Key ablations:

• Pointwise KL clipping. Without clipping, style tokens (e.g., wait, think) dominate the divergence and destabilize training — critical since OPSD converges within a few hundred steps.
• Generation-style pairing. Of four combinations of Qwen3 thinking modes, TM-off student + TM-on teacher produced the largest KL on math tokens and the best downstream results.

Where OPSD fails

*The Many Faces of On-Policy Distillation* (arXiv:2605.11182) maps the boundary. Success depends not on teacher strength but on whether the teacher's signal stays locally consistent on the student's own trajectories.

• Math reasoning: often fails. The privileged information (PI) is a sample-specific answer, unavailable at test time. OPSD effectively learns an average of many answer-conditioned teachers rather than a generalizable skill. On Qwen3-1.7B + OpenThoughts, answer-only PI, full-response PI, and even RL-trained teachers showed no stable gains on Math500/AIME24/25.
• System-prompt internalization, style/preference alignment: works well. Here PI is a shared latent rule, compressible into stable behavior that persists without PI at test time. On CharacterBench and EmotionBench, OPSD converges faster than GRPO/PPO with better sample efficiency.

Two concrete failure modes:

1. Student prefixes corrupt the teacher. Once the student takes a wrong branch, the teacher forced to continue from that state no longer behaves like a teacher solving from scratch. On GPQA-Diamond, a Qwen3-14B teacher solving independently scored 62.12%; conditioned on truncated Qwen3-1.7B student prefixes it dropped to 45.96% (−16.16 points). Of affected problems, 40 flipped right→wrong vs only 8 wrong→right. The model learns hedging (wait, maybe, backtracking), and response length balloons. 2. TopK reverse KL has biased gradients. TopK truncation breaks the constant-cancellation property of reverse KL, introducing bias that amplifies as the distribution drifts — loss, length, repetition, and accuracy all degrade (the model becomes verbose, then repeats "maybe" until the length cap, with accuracy near zero). Fixes: stop-gradient TopK, renormalized TopK, or sampled-token KL in policy-gradient form (the latter two stabilize training).

> Note: reverse KL (mode-seeking, conservative) is generally preferred over forward KL (mode-covering, which pushes students toward teacher-favored tokens with low student probability, causing sharper drift).

USD: a unified framework (Tencent, arXiv:2608.08176)

Vanilla OPSD makes two suboptimal defaults: uniform token weighting and full PI exposure. Real signal concentrates in few positions (the top 20% of tokens carry ~80% of gradient mass), and full PI can push the teacher beyond the student's support and leak the answer into supervision.

USD reformulates OPSD as constrained optimization: jointly choose token weights w and PI strength β under a student-capacity budget ε, maximizing learnable average divergence. A single dual variable λ — the marginal price of learning difficulty — simultaneously sets the token-inclusion threshold and PI adjustment, via a lightweight online primal-dual algorithm with only O(T) scalar updates per batch (no auxiliary networks, no extra forward passes).

Results (Qwen3, Avg@12 %):

| Method | 1.7B | 4B | 8B | Overall | |--------|------|----|----|---------| | Vanilla OPSD | 41.5 | 62.2 | 65.5 | 56.4 | | TIP (token only) | 43.0 | 63.0 | 66.7 | 57.1 | | PAINT (PI only) | 42.8 | 62.7 | 66.0 | 57.0 | | USD (joint) | 44.0 | 64.5 | 67.6 | 58.7 |

Overall +2.3 points; beats GRPO by +4.7 / +2.0 / +1.5 across scales with a far smaller sampling budget (1×1,024 tokens vs GRPO's 8×16k × 500 steps). Fixed hyperparameters: ε=0.3, KL clip 0.05, LoRA r64/α128, AdamW lr 5e-6, 300 steps. ε=0.15/0.5 on 1.7B drops to 42.0/41.5 — capacity must be matched.

Ecosystem

• DASD: entropy-routed teacher influence, preserving exploration.
• TRD: self-teacher rewrites entire rollouts, mitigating prefix failure.
• ROSD: reflection-guided, error-localized KL losses; better OOD generalization.
• Variants using world feedback (compiler errors, unit tests), visual/multimodal PI, and suffix-conditioned diffusion LLMs.
• Engineering: parameter sharing saves roughly 40–60% GPU memory vs external-teacher OPD.

When to use, when to avoid

Good fits: reasoning compression (shorter chains without accuracy loss, on strong ≥8B bases); system-prompt internalization and style/preference alignment; competition math with a strong, stable base.

Avoid when: PI is per-sample and unavailable at test time; the model is weak (<8B); or the student has already drifted badly (teacher corruption, length explosion).

Open questions

• Integrating verification signals (currently no explicit correctness check; distills only 2k–4k tokens).
• Population self-distillation (distilling a model's correct attempts into its wrong ones, removing the need for reasoning datasets).
• Curriculum learning: problems beyond the model's comprehension threshold can't be taught even with the answer.
• Whether larger token budgets remain effective for multi-turn, long-context planning.

Bottom line

OPSD manufactures an information-access asymmetry during training: the teacher's extra glance at the answer gets distilled into the student's parameters, saving that glance at inference. It is cheap, dense-signal, and teacher-free — but whether PI is a shared rule or a per-problem secret determines whether it is a cure or noise. USD pushes it from craft toward a principled framework.