Overview
Field: ML Authors: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song Date: 2026-07-02 arXiv: 2607.02502 Categories: cs.LG, cs.AI
Summary
On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization. It also introduces a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time.
The authors introduce DemoPSD, a novel framework that resolves these problems through the idea of *selective adoption of teacher guidance*. Instead of fitting the full teacher distribution, DemoPSD guides the student toward a *reverse-KL barycenter target*—a weighted geometric combination of the teacher distribution and the student distribution—naturally balancing learning from the teacher with preserving the student's own reasoning ability. The two distributions' disagreement is measured and used to adaptively control the mixing ratio at each token position.
Theoretical Guarantees
DemoPSD provably achieves:
1. Leakage attenuation: effectively mitigates privileged information leakage. 2. Exploration preservation: maintains exploration under dense token-level distillation.
Experimental Results
Extensive experiments on SciKnowEval across four scientific domains show that DemoPSD outperforms GRPO and SDPO, while maintaining higher training entropy and generalizing robustly on the out-of-distribution GPQA benchmark.
---
Source: arXiv:2607.02502