DemoPSD: Disagreement-Modulated Policy Self-Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-06

Summary

DemoPSD (arXiv:2607.02502) is a new framework for on-policy self-distillation (OPSD) in training large language models to reason. In OPSD, a single model acts as both teacher and student with different information access, but the teacher's dense token-level supervision conditioned on privileged information can cause overfitting to in-domain patterns, suppress exploration, and introduce privileged information leakage, where the student encodes answer-dependent shortcuts unavailable at test time. DemoPSD addresses this via selective adoption of teacher guidance: instead of fitting the full teacher distribution, the student is steered toward a reverse-KL barycenter target, a weighted geometric mixture of the teacher and student distributions that balances learning from the teacher with preserving the student's own reasoning. Distributional disagreement adaptively controls the mixture ratio at each token position. The authors theoretically show DemoPSD achieves leakage attenuation and exploration preservation. Experiments on SciKnowEval across four scientific domains show DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and robustly generalizing on the out-of-distribution GPQA benchmark.

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Field: Machine Learning Authors: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song Published: 2026-07-02 arXiv: 2607.02502 Categories: cs.LG, cs.AI

Background

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization. It also introduces a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time.

The DemoPSD Approach

DemoPSD resolves these problems through the idea of *selective adoption of teacher guidance*:

Instead of fitting the full teacher distribution, DemoPSD steers the student toward a *reverse-KL barycenter target* — a weighted geometric mixture of the teacher and student distributions, naturally balancing learning from the teacher with retaining the student's own reasoning ability.
The disagreement between the two distributions is measured and used to adaptively control the mixture ratio at each token position.

Theoretical Guarantees

The authors prove that DemoPSD achieves:

1. Leakage attenuation: effectively mitigating privileged information leakage. 2. Exploration preservation: maintaining exploration despite dense token-level distillation.

Experimental Results

Extensive experiments on SciKnowEval, spanning four scientific domains, show that DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and robustly generalizing on the out-of-distribution GPQA benchmark.

---

*Auto-collected on 2026-07-06*