[论文] DemoPSD: Disagreement-Modulated Policy Self-Distillation

论文概要

研究领域: ML 作者: Yunhe Li, Hao Shi, Wenhao Liu 发布时间: 2026-07-04 arXiv: 2507.03244

中文摘要

On-policy self-distillation (OPSD) 已成为训练大型语言模型（LLM）进行推理的实用方法，其中单一模型在不同信息访问层级下同时扮演教师和学生角色。然而，近期研究发现，教师基于特权信息（privileged information）的密集 token 级监督会导致模型过拟合域内模式、抑制探索能力，并损害跨域泛化能力，同时引出一个更根本的问题：特权信息泄露（privileged information leakage）——学生在训练时编码了测试时不可用的答案依赖捷径。

本文提出 DemoPSD，一种通过选择性采纳教师指导来解决上述问题的新框架。DemoPSD 不再拟合完整的教师分布，而是将学生引导至一个反向 KL 重心目标（reverse-KL barycenter target）——即教师分布与学生分布的加权几何组合，从而在向教师学习的同时保留学生自身的推理能力。该方法通过衡量两种分布的差异，自适应地控制每个 token 位置的混合权重。

理论证明 DemoPSD 实现了两个关键特性：(1) 泄露衰减（leakage attenuation）——有效缓解特权信息泄露；(2) 探索保留（exploration preservation）——在密集 token 级蒸馏下保持探索能力。在 SciKnowEval 四个科学领域的广泛实验表明，DemoPSD 在优于 GRPO 和 SDPO 的同时，保持了更高的训练熵，并对分布外 GPQA 基准展现出稳健的泛化能力。

原文摘要

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher wi...

--- *自动采集于 2026-07-05*

#论文 #arXiv #ML #小凯