[论文] RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcem...

论文概要

研究领域: NLP
作者: Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen
发布时间: 2026-09-17
arXiv: 2609.20784

中文摘要

用强化学习（RL）训练的多轮智能体每条轨迹只获得一个标量奖励，这促使自蒸馏在线策略蒸馏（OPD）方法的出现——从具有特权任务技能的自教师提供密集的 token 级监督，让无技能学生将其内化。然而，在智能体任务中有两个发现削弱了这一方案：仅凭特权信息并不总能使教师可靠，教师监督的效益是阶段依赖的。因此我们提出 RetireOPD（自退休在线策略蒸馏），首先用环境奖励优化一个解耦的、技能条件化教师，然后联合 RL 和 OPD 训练无技能学生。RetireOPD 不遵循预定义的蒸馏调度，而是采用自适应退休机制：当学生与教师的差异停止缩小且达到教师成功率的某个目标比例时，学生自行放弃教师，此后仅用 RL 训练。在 1.5B 到 7B 的 Qwen2.5 模型上，RetireOPD 将 ALFWorld 成功率比 RL 基线提高 14.1% 到 18.8%，WebShop 准确率提高 11.8% 到 19.0%，并在所有设置中超越了自己的技能条件化教师。

原文摘要

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retire...

自动采集于 2026-09-19

#论文 #arXiv #NLP #小凯