DemoPSD: Disagreement-Modulated Policy Self-Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-06

Summary

DemoPSD (arXiv:2607.02502) is a new framework for training large language models to reason via on-policy self-distillation (OPSD), where a single model serves as both teacher and student. Prior work showed that dense token-level teacher supervision conditioned on privileged information can cause overfitting to in-domain patterns, suppress exploration, hurt cross-domain generalization, and introduce privileged information leakage—students encode answer-dependent shortcuts unavailable at test time. DemoPSD addresses these issues through selective adoption of teacher guidance: rather than fitting the full teacher distribution, the student is guided toward a reverse-KL barycenter target, a weighted geometric combination of teacher and student distributions. Disagreement between the two distributions adaptively modulates the mixing weight at each token position. The authors theoretically prove that DemoPSD achieves leakage attenuation and preserves exploration under dense token-level distillation. Experiments on SciKnowEval across four scientific domains show DemoPSD outperforms GRPO and SDPO while maintaining higher training entropy and generalizing robustly on the out-of-distribution GPQA benchmark.

Overview

Research field: Machine Learning Authors: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song Published: 2026-07-02 arXiv: 2607.02502 Categories: cs.LG, cs.AI

Problem

On-policy self-distillation (OPSD) has become a practical method for training LLMs to reason, with a single model acting as both teacher and student at different levels of information access. However, the teacher's dense token-level supervision, conditioned on privileged information, can:

Cause overfitting to in-domain patterns
Suppress exploration
Hurt cross-domain generalization
Introduce privileged information leakage: the student encodes answer-dependent shortcuts that are unavailable at test time

DemoPSD Approach

DemoPSD resolves these problems through selective adoption of teacher guidance. Instead of fitting the full teacher distribution, the student is guided toward a reverse-KL barycenter target—a weighted geometric combination of the teacher and student distributions—naturally balancing learning from the teacher with preserving the student's own reasoning. The disagreement between the two distributions is measured and used to adaptively control the mixing weight at each token position.

Theoretical Guarantees

The paper proves that DemoPSD achieves:

1. Leakage attenuation: effectively mitigating privileged information leakage 2. Exploration preservation: maintaining exploration under dense token-level distillation

Experiments

Extensive experiments on SciKnowEval spanning four scientific domains show DemoPSD:

Outperforms GRPO and SDPO
Maintains higher training entropy
Generalizes robustly on the out-of-distribution GPQA benchmark

Original Abstract (excerpt)

> On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization... We introduce DemoPSD, a novel framework that resolves such problems through the idea of *selective adoption of teacher guidance*.

Paper: https://arxiv.org/abs/2607.02502