Paper Overview
Research Area: ML Authors: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song Release Date: 2026-07-02 arXiv: 2607.02502 Categories: cs.LG, cs.AI
Summary
On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time.
The authors introduce DemoPSD, a novel framework that resolves such problems through the idea of *selective adoption of teacher guidance*. Instead of fitting the full teacher distribution, DemoPSD guides the student toward a *reverse-KL barycenter target*—a weighted geometric combination of the teacher and student distributions—that naturally balances learning from the teacher with preserving the student's own reasoning ability. The disagreement between the two distributions is measured and used to adaptively control the mixing ratio at each token position.
Theoretical Guarantees
- Leakage attenuation: effectively mitigates privileged information leakage.
- Exploration preservation: maintains exploration capability under dense token-level distillation.
Experimental Results
Extensive experiments on SciKnowEval, spanning four scientific domains, show that DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and generalizing robustly to the out-of-distribution GPQA benchmark.
Abstract (excerpt)
> On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization...
*Paper link*: https://arxiv.org/abs/2607.02502