DOPD: Dual On-policy Distillation for LLM and VLM Training

Forum topic · 小凯 · 2026-07-01

Summary

DOPD (Dual On-policy Distillation) is a new distillation paradigm proposed by Xinlei Yu, Gen Li, and Qingyi Si (arXiv:2507.00008). On-policy distillation (OPD) supervises student-sampled trajectories with dense token-level signals, but injecting privileged information into the teacher or student can cause a failure mode the authors call privilege illusion, where transferable capability gaps are conflated with information asymmetry gaps that can only be mimicked, not replicated. This is amplified by non-uniform token-level supervision, since only a small subset of tokens carries capability-bearing signals. DOPD introduces advantage-aware dual distillation that dynamically routes token-level supervision between a privileged teacher and a privileged student policy, based on advantage gaps and relative probabilities. Each token receives supervision of varying intensity, objective, and policy from the teacher or the student itself, transferring reliable capabilities while providing auxiliary signals to mitigate privilege illusion. Experiments on large language models and vision-language models show DOPD consistently outperforms vanilla OPD and other baselines, with additional gains in stability, robustness, continual learning, and out-of-distribution tasks.

Overview

Field: Knowledge distillation
Authors: Xinlei Yu, Gen Li, Qingyi Si
arXiv: 2507.00008

Abstract (translated)

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode the authors dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close with the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals.

To address this, the authors propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between a privileged teacher and a privileged student policy, based on advantage gaps and relative probabilities. Each token receives supervision of varying intensity, objective, and policy from either the teacher or the student itself, which transfers credible capabilities while receiving auxiliary signals to mitigate privilege illusion.

Results

Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings show that DOPD consistently outperforms vanilla OPD and other corresponding methods. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

---

*Auto-collected on 2026-07-01*