DemoPSD: Disagreement-Modulated Policy Self-Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-05

Summary

DemoPSD (arXiv:2507.03244) is a new framework for on-policy self-distillation (OPSD) in training large language models for reasoning. In OPSD, a single model acts as both teacher and student with different information access levels. Prior work showed that dense token-level teacher supervision conditioned on privileged information causes overfitting to in-domain patterns, suppresses exploration, harms cross-domain generalization, and introduces privileged information leakage—students encode answer-dependent shortcuts unavailable at test time. DemoPSD addresses these issues through selective adoption of teacher guidance: instead of fitting the full teacher distribution, it steers the student toward a reverse-KL barycenter target, a weighted geometric combination of teacher and student distributions. Mixing weights at each token position are adapted based on disagreement between the two distributions. The authors theoretically prove two properties: leakage attenuation and exploration preservation. Experiments across four scientific domains of SciKnowEval show DemoPSD outperforms GRPO and SDPO, maintains higher training entropy, and generalizes robustly to the out-of-distribution GPQA benchmark. Authors: Yunhe Li, Hao Shi, Wenhao Liu.

Paper Overview

Research area: Machine Learning
Authors: Yunhe Li, Hao Shi, Wenhao Liu
arXiv: 2507.03244

Background

On-policy self-distillation (OPSD) has become a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to:

Overfitting to in-domain patterns
Suppressed exploration
Hurt cross-domain generalization

It also introduces a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time.

DemoPSD

DemoPSD resolves these problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target—a weighted geometric combination of the teacher and student distributions—naturally balancing learning from the teacher with preserving the student's own reasoning capability. The method adaptively controls the mixing weight at each token position by measuring the disagreement between the two distributions.

Theoretical Guarantees

The authors prove two key properties:

1. Leakage attenuation — effectively mitigates privileged information leakage 2. Exploration preservation — maintains exploration under dense token-level distillation

Experiments

Extensive experiments across four scientific domains of SciKnowEval show that DemoPSD:

Outperforms GRPO and SDPO
Maintains higher training entropy
Generalizes robustly to the out-of-distribution GPQA benchmark

---

*Auto-collected on 2026-07-05*