DemoPSD: Disagreement-Modulated Policy Self-Distillation

论文概要

研究领域: ML
作者: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
发布时间: 2026-07-02
arXiv: 2607.02502
分类: cs.LG, cs.AI

中文摘要

在线策略自蒸馏（OPSD）已成为训练大语言模型（LLM）进行推理的实用方法，其中单一模型同时扮演教师和学生角色，但拥有不同级别的信息访问权限。然而，近期研究发现，教师基于特权信息的密集token级监督会导致对领域内模式的过拟合、抑制探索能力，并损害跨域泛化性能。更根本的问题是特权信息泄露：学生编码了在测试时不可用的答案依赖捷径。本文提出DemoPSD，一种通过选择性采纳教师指导来解决上述问题的新框架。与拟合完整教师分布不同，DemoPSD将学生引导至一个反向KL重心目标——即教师分布与学生分布的加权几何组合，从而自然平衡向教师学习与保留学生自身推理能力。我们度量两者分布的差异，并利用这种差异在每个token位置自适应控制混合比例。理论上证明，DemoPSD实现了**(1)** 泄露衰减：有效缓解特权信息泄露；(2) 探索保留：在密集token级蒸馏下保持探索能力。在横跨四个科学领域的SciKnowEval上的大量实验表明，DemoPSD优于GRPO和SDPO，同时保持更高的训练熵，并在分布外的GPQA基准上稳健泛化。

原文摘要

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher...

自动采集于 2026-07-06

#论文 #arXiv #ML #小凯