Relay-OPD: Trajectory-Relayed On-Policy Distillation for LLM Knowledge Transfer

Forum topic · ✨步子哥 · 2026-07-29

Summary

Relay-OPD, presented by researchers from Zhejiang University and Alibaba's Yuvion team, introduces a novel trajectory-level intervention method for on-policy distillation (OPD) that addresses structural prefix failures in student LLMs. The core insight is an asymmetry between teacher and student models: when a reasoning trajectory goes off course, teachers tend to generate reflective tokens such as "But" or "Wait" to change direction, while students tend to continue with tokens like "So" or "Therefore." This naturally occurring divergence allows a handoff trigger that asks the teacher to temporarily take over the trajectory, correct course, and return control, with results showing significant gains in both accuracy and output efficiency on mathematical reasoning benchmarks.

Overview

Relay-OPD (Relay On-Policy Distillation) is a trajectory-level intervention strategy for on-policy distillation (OPD), presented in the paper "Pass the Baton: Trajectory-Relayed On-Policy Distillation" (arXiv: 2607.26057) by Haolei Xu, Xiaowen Xu, Haiwen Hong, and colleagues from Zhejiang University and Alibaba's Yuvion team. The method targets prefix failures in student-generated reasoning chains and achieves large accuracy gains while substantially shortening training trajectories.

arXiv: https://arxiv.org/abs/2607.26057
Code: https://github.com/zju-real/Relay-OPD
Project page: https://zju-real.github.io/Relay-OPD

The Problem: Prefix Failure in OPD

On-policy distillation trains a small student LLM using token-level supervision from a larger teacher on the student's own generated trajectories. A structural weakness is the prefix failure: once the student takes an early wrong turn, every subsequent token is built on a flawed foundation. Earlier fixes have structural shortcomings:

Fixed-length truncation (ESR, FastOPD): Cuts at an arbitrary length; may chop good continuations or fail to stop accumulated errors.
Offline rewriting (TRD): Rewrites trajectories after the fact, leaving visible "patched" artifacts that lose natural reasoning rhythm.
Token-level blending (SKD): Mixes teacher/student distributions but cannot break the student's already-locked repetitive patterns.

Core Insight: Teacher-Student Continuation Asymmetry

On a failing prefix, the teacher tends to reflect (e.g., tokens such as But, Wait, However), while the student tends to persist (e.g., So, Therefore). This divergence needs no external label or verifier; it is detectable from the models' own next-token distributions.

The handoff criterion is:

$$\phi(h) = \mathbb{1}[a^T(h) \in \mathcal{R}] \cdot \mathbb{1}[\mathcal{K}_S(h) \cap \mathcal{R} = \emptyset]$$

where $a^T(h)$ is the teacher's top-1 token, $\mathcal{K}_S(h)$ is the student's top-K support set, and $\mathcal{R}$ is a predefined reflection-word set (with casing and leading-space variants). The trigger fires precisely when the teacher wants to turn around and the student does not.

The Relay Mechanism

When the trigger fires:

1. Student runs its leg up to the handoff point. 2. Teacher takes over and generates a short reflective correction. 3. Control returns to the student for the rest of the trajectory. 4. The mixed trajectory is used as standard OPD training data.

A relay budget (typically 2-3 interventions per trajectory) keeps the trajectory mostly student-generated, preserving OPD's distribution-matching advantage while concentrating intervention at early, high-leverage decision points. Execution uses a speculative-decoding engine (§3.3) that shares a single inference path and switches model parameters, keeping relay cost near single-model inference.

Experimental Results

Setup: teacher = Qwen3-4B-Instruct-2507; students = Qwen3-0.6B and Qwen3-1.7B (Non-Thinking). Eight math reasoning benchmarks: AIME24/25/26, MATH, AMC23, Olympiad Bench, HMMT Feb26, HMMT Nov25.

Results on Qwen3-1.7B student:

| Method | Avg. Accuracy | Training Trajectory Length | | --- | --- | --- | | Student baseline | 24.84 | — | | SFT | 33.20 | 4262 | | KD | 33.75 | 4262 | | GRPO | 34.42 | 2558 | | OPD | 41.23 | 4658 | | TRD | 30.69 | 2785 | | FastOPD | 45.47 | 2709 | | SKD | 42.35 | 4753 | | Relay-OPD | 46.96 | 2296 |

Key findings:

+5.73 over OPD and +1.49 over the strongest baseline FastOPD.
Training trajectory length drops from 4658 to 2296, a 50.7% reduction.
On AIME25 and AIME26, Relay-OPD exceeds OPD by +7.29 and +7.19 respectively.
On the 0.6B student, +3.01 over OPD.
Versus FastOPD on AIME25/AIME26/HMMT Feb26, response length is 17.9%/14.2%/28.3% shorter while accuracy is 2.39/4.17/1.14 points higher.

Why TRD and SKD underperform: TRD rewrites leave visible edit artifacts that disrupt natural reasoning rhythm (Appendix E.2). SKD cannot escape the student's locked repetitive loops (Appendix E.3). Relay-OPD intervenes at the directional-divergence moment with a clean restart signal.

Limitations

1. The reflection-word set $\mathcal{R}$ is hand-defined; transfer to non-math domains (code, dialogue) is open. 2. Validated only on math reasoning benchmarks. 3. Relay budget is a fixed hyperparameter (2-3); no adaptive scheme by task difficulty. 4. The teacher-student asymmetry is empirically validated on Qwen3; other model families may differ in reflective behavior, which would weaken the trigger.

Takeaway

Relay-OPD reframes distillation supervision: instead of supervising uniformly at every token (OPD) or fixing things up after the fact (TRD), supervision should concentrate at the directional turning points of a reasoning trajectory. The deeper lesson: weak models fail less from a lack of knowledge and more from not knowing when to change direction.