[论文] Pass the Baton: Trajectory-Relayed On-Policy Distillation

论文概要

研究领域: NLP
作者: Haolei Xu, Xiaowen Xu, Haiwen Hong, Zixuan Ni, Hongxing Li, Yiwen Qiu, Weiming Lu, Yongliang Shen
发布时间: 2026-07-28
arXiv: 2607.26057

中文摘要

在线策略蒸馏（OPD）将token级监督建立在学生模型自身的生成轨迹上，但存在一个前缀失效问题：一旦学生模型在推理方向上出现偏差，后续所有生成都会基于这个错误的方向继续，产生被误导的延续内容，从而得到不可靠的监督信号并浪费计算资源。我们发现，在失败前缀上存在一种教师-学生延续的不对称性——教师倾向于纠正方向而学生则继续沿原方向前进，并将这一现象转化为Relay-OPD中的无标签交接触发机制。在训练过程中，Relay-OPD通过在检测到的触发点让教师模型短暂接管来构建中继轨迹，生成一段教师引导的序列，之后学生模型恢复并基于该轨迹进行优化。有限的中继预算将干预集中在关键早期位置，同时限制与原始学生策略的偏离。在Qwen3-4B-Instruct-2507作为教师、Qwen3-0.6B/1.7B-Non-Thinking作为学生的八个数学推理基准测试中，Relay-OPD在每个基准上都取得了最佳或次佳结果，对于1.7B模型平均比标准OPD提升+5.73%，比最强基线FastOPD提升+1.49%，0.6B模型也有持续提升。训练轨迹长度减少超过50%。

原文摘要

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the result...

自动采集于 2026-07-30

#论文 #arXiv #NLP #小凯