RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning (arXiv 2609.20784)

Forum topic · 小凯 · 2026-09-19

Summary

RetireOPD is a training framework for multi-turn agents that combines reinforcement learning (RL) with on-policy distillation (OPD) and an adaptive retirement mechanism. Multi-turn RL agents typically receive only one scalar reward per trajectory, and self-distillation from a privileged teacher offers dense token-level supervision, but two issues undermine this: privileged information does not always make a teacher reliable, and teacher supervision benefits are stage-dependent. RetireOPD first optimizes a decoupled, skill-conditioned teacher with environment rewards, then trains a skill-free student jointly with RL and OPD. Instead of a predefined schedule, an adaptive retire mechanism lets the student abandon the teacher once the student-teacher gap stops shrinking and the student reaches a target fraction of teacher success rate, after which training continues with RL alone. On Qwen2.5 models from 1.5B to 7B parameters, RetireOPD improves ALFWorld success rates by 14.1% to 18.8% over RL baselines and WebShop accuracy by 11.8% to 19.0%, surpassing even its own skill-conditioned teacher in all settings. Paper: arXiv 2609.20784, posted September 17, 2026, authors from Zhejiang University-affiliated team including Yan Yu, Zhengxi Lu, and Yongliang Shen.

Paper Overview

Field: NLP Authors: Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen Published: 2026-09-17 arXiv: 2609.20784

Introduction

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them.

However, this recipe is undermined by two findings in agentic tasks:

1. Privileged information alone does not always make a teacher reliable. 2. The benefit of teacher supervision is stage-dependent.

Method: RetireOPD

RetireOPD (Self-Retiring On-Policy Distillation) works in two phases:

Phase 1: Optimize a decoupled, skill-conditioned teacher with environment rewards.
Phase 2: Train a skill-free student jointly with RL and OPD.

Rather than following a predefined distillation schedule, RetireOPD adopts an Adaptive Retire mechanism: when the gap between the student and teacher stops shrinking and the student reaches a target proportion of the teacher's success rate, the student retires the teacher and continues training with RL only.

Results

Experiments on Qwen2.5 models (1.5B to 7B) show:

ALFWorld: success rate improved by 14.1% to 18.8% over the RL baseline.
WebShop: accuracy improved by 11.8% to 19.0%.
RetireOPD surpasses its own skill-conditioned teacher in all settings.

---

*Auto-collected on 2026-09-19*