Relay-OPD: Trajectory-Relayed On-Policy Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-30

Summary

Relay-OPD (Relay On-Policy Distillation) addresses the prefix failure problem in on-policy distillation (OPD), where a student LLM that commits to a wrong reasoning direction generates misdirected continuations that yield unreliable supervision and waste compute. The authors observe a teacher-student continuation asymmetry on failed prefixes: the teacher model tends to redirect the reasoning while the student continues along the original path. This asymmetry is converted into a label-free handoff trigger. During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points, producing a teacher-guided leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates interventions at critical early positions while limiting divergence from the student's original policy. Using Qwen3-4B-Instruct-2507 as teacher and Qwen3-0.6B/1.7B-Non-Thinking as students across eight math reasoning benchmarks, Relay-OPD achieves best or second-best results on every benchmark: for the 1.7B model, an average gain of +5.73% over standard OPD and +1.49% over FastOPD, with consistent improvements for the 0.6B model and over 50% reduction in training trajectory length. Paper: arXiv 2607.26057.

Paper Overview

Field: NLP Authors: Haolei Xu, Xiaowen Xu, Haiwen Hong, Zixuan Ni, Hongxing Li, Yiwen Qiu, Weiming Lu, Yongliang Shen arXiv: 2607.26057

Abstract

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute.

The authors identify a teacher-student continuation asymmetry on failed prefixes: the teacher tends to redirect the reasoning, while the student continues along the original direction. They convert this phenomenon into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD).

Method

During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates interventions at critical early positions while limiting divergence from the original student policy.

Results

Experiments use Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B/1.7B-Non-Thinking as students across eight math reasoning benchmarks:

Best or second-best results on every benchmark
For the 1.7B model: +5.73% average improvement over standard OPD and +1.49% over the strongest baseline FastOPD
Consistent improvements for the 0.6B model
Training trajectory length reduced by more than 50%