On-Policy Distillation: The New Post-Training Paradigm and How Four Major Labs Engineer It Differently

Forum topic · 小凯 · 2026-05-02

Summary

This report analyzes On-Policy Distillation (OPD), an emerging post-training technique where a student model generates its own trajectories while a teacher provides dense per-token signals via Reverse KL divergence. It explains why pure RL suffers from low signal density (one reward per long reasoning chain) and how OPD combines RL's on-policy sampling with distillation's token-level supervision at roughly 50-100x RL's signal density. The article contrasts how Qwen3, GLM-5, MiMo, and DeepSeek V4 implement OPD differently: token-level vs full-vocabulary KL, pure distillation vs KL+ORM mixtures, teacher selection (same-architecture checkpoints, multi-expert routing, 10+T-parameter heterogeneous experts, cross-scale distillation), and pipeline placement. Benchmarks show OPD matching or exceeding RL at ~1/10 the compute, students surpassing teachers, and strong results on AIME, SWE-bench, and LMArena. Key conclusions: OPD complements rather than replaces RL, Reverse KL's mode-seeking property prevents catastrophic forgetting, and the shift from parameter-space to logit-space knowledge integration may be post-training's most important paradigm change.

Overview

In early 2025, DeepSeek R1 showed that GRPO-based RL can let models develop reasoning abilities on their own. But the industry quickly hit a ceiling: pure RL has fundamentally low signal density — a 500-token reasoning chain yields only a single scalar reward, so per-token learning signal is ~1/500. SFT and off-policy distillation have dense per-token signals but suffer from distribution mismatch (exposure bias). On-Policy Distillation (OPD) combines the best of both: the student generates its own responses (on-policy) while the teacher scores every token (dense signal).

Key points

Core method: OPD minimizes Reverse KL — D_KL(π_θ || π_teacher) — on student-generated trajectories. Reverse KL is mode-seeking (students focus on the teacher's highest-probability solutions) rather than mean-seeking, which suits verifiable-reasoning tasks.
One-line change: In a GRPO framework, replace the group-normalized advantage A_i = (r_i - mean_group) / std_group with the per-token teacher-student log-ratio A_t = stop_grad[log(π_teacher(y_t|x,y_<t) / π_θ(y_t|x,y_<t))] — what ThinkingMachines.ai calls "a one-line change."

How four labs diverge on the same formula

1. KL granularity: Qwen3, GLM-5, and MiMo use token-level KL (Monte Carlo approximation, cheap). DeepSeek V4 uses full-vocabulary KL, criticizing token-level as "high-variance, unstable," and built three infrastructure layers: ZeRO-like teacher weight scheduling, last-layer hidden-state caching, and custom TileLang kernels. 2. Reward mixing: GLM-5 and DeepSeek V4 use pure distillation (KL only). MiMo's MOPD adds an ORM term (KL + α·ORM); its ablation shows pure ORM RL < MOPD w/o ORM < full MOPD. 3. Teacher choice: GLM-5 uses earlier checkpoints of the same architecture; MiMo routes tasks across domain experts plus SFT and self-distillation teachers; DeepSeek V4 integrates 10+ trillion-parameter heterogeneous experts with per-expert weighting; Qwen3 distills across scales (235B-A22B/32B teachers → 0.6B–30B students). 4. Pipeline position: Qwen3 uses OPD as a standalone sub-pipeline replacing full RL for efficiency; GLM-5 as a final recovery stage against catastrophic forgetting; MiMo as the main third stage for multi-expert integration; DeepSeek V4 as a unification stage compressing 10+ experts into one model.

Results

ThinkingMachines benchmark (AIME'24, from the same off-policy checkpoint): off-policy distillation 60% (1× compute), pure RL 67.6% (10×), OPD 74.4% (1×).
Qwen3: Qwen3-30B-A3B matches QwQ-32B reasoning; the six-size lightweight series trained with ~1/10 the GPU time of flagship RL.
GLM-5: topped LMArena text and code leaderboards; Humanity's Last Exam 50.4, SWE-bench Verified 77.8, Terminal-Bench 2.0 56.2 — at or above Claude Opus 4.5.
MiMo: students exceed their strongest teachers (AIME 2025 +0.2, HMMT Feb 2025 +1.8, LiveCodeBench +0.6, HLE +0.9) — an emergent effect of multi-teacher signals.

Why OPD works

Logit space beats parameter space: weight merging and mixed RL interfere in parameter space; OPD moves integration to logit space, where experts contribute knowledge independently.
Reverse KL + on-policy = resistance to forgetting: the student only aligns with the teacher on its own trajectories, never forced to cover teacher distributions in other domains.
Signal density: estimated 50–100× that of pure RL, since every token carries a learning signal.

Directions for 2026

Full-vocabulary KL becoming standard as infrastructure (e.g., TileLang) is open-sourced.
Replacing ORM with PRM (process reward models) for even denser signals: teacher logits + PRM + ORM.
Iterative co-evolution: distilled students re-enter expert RL to become next-generation teachers, akin to AlphaZero self-play.
Inference-time distillation: querying teacher distributions at critical "forking tokens" during beam search or sampling.

Conclusions

1. OPD supplements rather than replaces RL — its role is consolidating RL-learned abilities, not teaching new ones. 2. Reverse KL's mode-seeking property is the root cause of OPD's forgetting resistance. 3. The four labs' divergences reflect different goals: efficiency (Qwen3), recovery (GLM-5), integration (MiMo), compression (DeepSeek V4). 4. The biggest engineering challenge is infrastructure, not algorithms. 5. The shift from parameter-space to logit-space integration may be post-training's most important paradigm change.

*Core references: Qwen3 Technical Report (2505.09388), GLM-5 (2602.15763), MiMo-V2-Flash (2601.02780), DeepSeek-V4 Technical Report, arXiv 2604.13016 "Rethinking OPD", arXiv 2603.25562 "Empirical Failure Modes", ThinkingMachines.ai OPD blog.*