DemoPSD: Disagreement-Modulated Policy Self-Distillation

论文概要

研究领域: ML 作者: Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song 发布时间: 2026-07-02 arXiv: 2607.02502 分类: cs.LG, cs.AI

中文摘要

在线策略自蒸馏（OPSD）已成为训练大语言模型（LLM）进行推理的实用方法，其中单一模型同时扮演教师和学生角色，但拥有不同级别的信息访问权限。然而，近期研究发现，教师基于特权信息的密集token级监督会导致对领域内模式的过拟合、抑制探索能力，并损害跨域泛化性能。更根本的问题是*特权信息泄露*：学生编码了在测试时不可用的答案依赖捷径。本文提出DemoPSD，一种通过*选择性采纳教师指导*来解决上述问题的新框架。与拟合完整教师分布不同，DemoPSD将学生引导至一个*反向KL重心目标*——即教师分布与学生分布的加权几何组合，从而自然平衡向教师学习与保留学生自身推理能力。我们度量两者分布的差异，并利用这种差异在每个token位置自适应控制混合比例。理论上证明，DemoPSD实现了(1) *泄露衰减*：有效缓解特权信息泄露；(2) *探索保留*：在密集token级蒸馏下保持探索能力。在横跨四个科学领域的SciKnowEval上的大量实验表明，DemoPSD优于GRPO和SDPO，同时保持更高的训练熵，并在分布外的GPQA基准上稳健泛化。

原文摘要

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: *privileged information leakage*, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of *selective adoption of teacher guidance*. Instead of fitting the full teacher...

--- *自动采集于 2026-07-06*

#论文 #arXiv #ML #小凯