Relay-OPD: Trajectory-Relayed On-Policy Distillation for LLM Reasoning

Forum topic · 小凯 · 2026-07-30

Summary

Relay-OPD is a new on-policy distillation (OPD) method for training small language models on math reasoning tasks. Standard OPD suffers from a prefix failure problem: once a student model commits to a wrong reasoning direction, all subsequent token-level supervision is built on that deviation, yielding unreliable signals and wasted compute. The authors observe a teacher-student continuation asymmetry on failed prefixes—teachers tend to redirect reasoning while students persist— and convert this into a label-free handoff trigger. Relay-OPD constructs relay trajectories: at detected trigger points the teacher briefly takes over to produce a teacher-guided segment, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting deviation from the student's policy. With Qwen3-4B-Instruct-2507 as teacher and Qwen3-0.6B/1.7B-Non-Thinking as students, Relay-OPD achieved best or second-best results on eight math reasoning benchmarks, improving over standard OPD by +5.73% and over FastOPD by +1.49% for the 1.7B model, while reducing training trajectory length by over 50%.

Paper Overview

Field: NLP Authors: Haolei Xu, Xiaowen Xu, Haiwen Hong, Zixuan Ni, Hongxing Li, Yiwen Qiu, Weiming Lu, Yongliang Shen arXiv: 2607.26057

The Problem: Prefix Failure in On-Policy Distillation

On-policy distillation (OPD) grounds token-level supervision in the student model's own generated trajectories. However, OPD suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute.

Key Insight: Teacher-Student Continuation Asymmetry

The authors identify an asymmetry on failed prefixes: the teacher model tends to redirect the reasoning, while the student continues along the original (wrong) direction. This phenomenon is converted into a label-free handoff trigger in Relay-OPD.

Method: Relay Trajectories

At detected trigger points, the teacher briefly takes over to produce a teacher leg of the trajectory.
The student then resumes generation, and optimization is performed on the resulting relay trajectory.
A limited relay budget concentrates interventions on critical early positions while limiting deviation from the original student policy.

Results

Experimental setup: Qwen3-4B-Instruct-2507 as teacher; Qwen3-0.6B / 1.7B Non-Thinking as students; eight math reasoning benchmarks.

Best or second-best results on every benchmark.
For the 1.7B student: +5.73% average over standard OPD, +1.49% over the strongest baseline FastOPD.
Consistent gains for the 0.6B student.
Training trajectory length reduced by over 50%.

Original Abstract (excerpt)

> On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD)...

*Auto-collected on 2026-07-30*