DemoPSD: Disagreement-Modulated Policy Self-Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-05

Summary

DemoPSD (arXiv:2507.03244) is a new framework for on-policy self-distillation (OPSD) in training large language models for reasoning. In OPSD, a single model acts as both teacher and student with different information access levels, but the teacher's dense token-level supervision based on privileged information can cause overfitting to in-domain patterns, suppress exploration, and introduce privileged information leakage, where the student encodes answer-dependent shortcuts unavailable at test time. DemoPSD addresses this via selective adoption of teacher guidance: instead of fitting the full teacher distribution, it steers the student toward a reverse-KL barycenter target—a weighted geometric combination of teacher and student distributions—with mixing weights adaptively controlled per token by distribution disagreement. The authors prove two properties: leakage attenuation and exploration preservation. Experiments on four scientific domains of SciKnowEval show DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and generalizing robustly to the out-of-distribution GPQA benchmark.

Paper Overview

Research Area: Machine Learning Authors: Yunhe Li, Hao Shi, Wenhao Liu Published: 2026-07-04 arXiv: 2507.03244

Background

On-policy self-distillation (OPSD) has become a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to:

Overfitting to in-domain patterns
Suppression of exploration
Hurt cross-domain generalization
Privileged information leakage — the student encodes answer-dependent shortcuts that are unavailable at test time

The DemoPSD Approach

DemoPSD resolves these problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target: a weighted geometric combination of the teacher and student distributions that naturally balances learning from the teacher while preserving the student's own reasoning ability.

The method measures the disagreement between the two distributions and adaptively controls the mixing weight at each token position.

Theoretical Guarantees

The paper proves two key properties:

1. Leakage attenuation — effectively mitigates privileged information leakage 2. Exploration preservation — maintains exploration under dense token-level distillation

Experimental Results

Extensive experiments on four scientific domains of SciKnowEval show that DemoPSD:

Outperforms GRPO and SDPO
Maintains higher training entropy
Shows robust generalization to the out-of-distribution GPQA benchmark

---

*Auto-collected on 2026-07-05*