DemoPSD: Disagreement-Modulated Policy Self-Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-05

Summary

DemoPSD (arXiv:2507.03244) is a new framework for on-policy self-distillation (OPSD) in training large language models for reasoning. In OPSD, a single model serves as both teacher and student with different levels of information access, but dense token-level supervision conditioned on privileged information can cause overfitting, suppressed exploration, and privileged information leakage—where the student encodes answer-dependent shortcuts unavailable at test time. DemoPSD addresses this via selective adoption of teacher guidance: instead of fitting the full teacher distribution, it steers the student toward a reverse-KL barycenter target, a weighted geometric combination of teacher and student distributions, with per-token mixing weights adaptively controlled by the disagreement between the two distributions. The authors theoretically prove that DemoPSD achieves leakage attenuation and exploration preservation. Experiments on four scientific domains of SciKnowEval show DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and robust generalization to the out-of-distribution GPQA benchmark.

Overview

Field: Machine Learning
Authors: Yunhe Li, Hao Shi, Wenhao Liu
Published: 2026-07-04
arXiv: 2507.03244

Background

On-policy self-distillation (OPSD) has become a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to:

Overfitting to in-domain patterns
Suppressed exploration
Hurt cross-domain generalization
A more fundamental issue: privileged information leakage — the student encodes answer-dependent shortcuts that are unavailable at test time

DemoPSD

DemoPSD resolves these problems through selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target — a weighted geometric combination of the teacher and student distributions — that naturally balances learning from the teacher while preserving the student's own reasoning ability. The mixing weight at each token position is adaptively controlled by measuring the disagreement between the two distributions.

Theoretically, DemoPSD provably achieves two key properties:

1. Leakage attenuation — effectively mitigating privileged information leakage 2. Exploration preservation — retaining exploration under dense token-level distillation

Results

Extensive experiments across four scientific domains of SciKnowEval show that DemoPSD:

Outperforms GRPO and SDPO
Maintains higher training entropy
Exhibits robust generalization to the out-of-distribution GPQA benchmark

---

*Auto-collected on 2026-07-05.*