DOPD: Dual On-policy Distillation with Advantage-Aware Token Routing

Forum topic · 小凯 · 2026-07-01

Summary

This paper introduces DOPD (Dual On-policy Distillation), a distillation method for large language models (LLMs) and vision-language models (VLMs). On-policy distillation (OPD) supervises student-sampled trajectories with dense token-level signals, enabling strong capacity transfer. A common strategy to improve OPD is injecting privileged information into the teacher or the student, but the authors identify a failure mode called privilege illusion: conflating the transferable capability gap students should close with an information asymmetry gap that can be mimicked but never replicated. Token-level supervision's inherent non-uniformity—where only a small subset of tokens carries capability-bearing signals—amplifies this problem. DOPD addresses it with an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between a privileged teacher and a privileged student policy based on advantage gaps and relative probabilities. Each token receives supervision of varying intensity, objective, and policy, transferring trustworthy capabilities while using auxiliary signals to mitigate privilege illusion. Experiments on LLM and VLM settings show DOPD consistently outperforms vanilla OPD and comparable methods, with additional results confirming advantages in stability, robustness, continual learning, and out-of-distribution tasks. Paper: arXiv:2507.00008.

Paper Overview

Research area: Knowledge distillation
Authors: Xinlei Yu, Gen Li, Qingyi Si
Published: 2026-07-01
arXiv: 2507.00008

Key Contributions

On-policy distillation (OPD) supervises student-sampled trajectories with dense token-level signals, offering superior capacity transfer compared to off-policy alternatives.
A natural way to raise the performance frontier of OPD is injecting privileged information into the teacher or the student itself, but this introduces a failure mode the authors call privilege illusion: conflating the transferable capability gap students are meant to close with an information asymmetry gap that can only be mimicked, never replicated.
The problem is amplified by the non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals.
DOPD is proposed as an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between a privileged teacher and a privileged student policy, based on advantage gaps and relative probabilities.
Each token receives supervision with different intensity, objective, and policy from the teacher or the student itself—transferring trustworthy capabilities while auxiliary signals mitigate privilege illusion.
Extensive experiments on LLM and VLM settings show DOPD consistently outperforms vanilla OPD and other corresponding methods. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.

Original Abstract (excerpt)

> On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose ...

*Auto-collected on 2026-07-01*