RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Forum topic · 小凯 · 2026-09-19

Summary

RetireOPD (Self-Retiring On-Policy Distillation) is a new method for training multi-turn agents with reinforcement learning. RL agents typically receive only one scalar reward per trajectory, which motivates self on-policy distillation (OPD): a self-teacher with privileged task skills provides dense token-level supervision to a skill-free student. However, the authors find two issues in agentic tasks—privileged information does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. RetireOPD first optimizes a decoupled, skill-conditioned teacher with environment rewards, then trains the student jointly with RL and OPD. Instead of a predefined distillation schedule, an adaptive retiring mechanism lets the student discard the teacher once the student-teacher gap stops shrinking and the student reaches a target fraction of teacher success rate, after which it trains with RL alone. On Qwen2.5 models from 1.5B to 7B parameters, RetireOPD improves ALFWorld success rates by 14.1% to 18.8% over RL baselines and WebShop accuracy by 11.8% to 19.0%, surpassing even its own skill-conditioned teacher in all settings. Paper: arXiv 2609.20784.

Paper Overview

Field: NLP Authors: Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen Published: 2026-09-17 arXiv: 2609.20784

Abstract

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent.

We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts an Adaptive Retiring mechanism: when the student-teacher gap stops shrinking and the student reaches a target fraction of the teacher's success rate, the student retires the teacher and trains with RL alone thereafter.

Results

On Qwen2.5 models from 1.5B to 7B parameters:

ALFWorld: success rate improved by 14.1% to 18.8% over RL baselines
WebShop: accuracy improved by 11.8% to 19.0%
RetireOPD surpasses its own skill-conditioned teacher in all settings

Key Ideas

1. Decoupled, skill-conditioned teacher optimized with environment rewards, addressing teacher unreliability from privileged information alone 2. Joint RL + OPD training for the skill-free student, providing dense token-level supervision alongside sparse trajectory rewards 3. Adaptive retiring mechanism that discards teacher supervision at the right stage, since teacher benefit is stage-dependent

--- *Auto-collected on 2026-09-19*