Pass the Baton: When Students Drift Off Course, Teachers Briefly Take the Wheel (Relay-OPD)

Forum topic · ✨步子哥 · 2026-08-03

Summary

Relay-OPD (Trajectory-Relayed On-Policy Distillation) is a method from Zhejiang University and Alibaba's Yuvion team that fixes a structural flaw in on-policy distillation called prefix failure: once a small student model takes an early wrong turn in its reasoning chain, all subsequent tokens build on that deviation, and token-level feedback on the flawed trajectory becomes unreliable. The key insight is an asymmetry between teacher and student continuations: on failing prefixes, teachers tend to emit reflection tokens (But, Wait, However) while students push forward (So, Therefore). Relay-OPD detects this divergence online—without external verifiers—and triggers a handoff where the teacher briefly generates a corrective segment before returning control to the student, under a limited relay budget. Implemented in a speculative decoding engine, the method raises Qwen3-1.7B student accuracy to 46.96% across eight math benchmarks (+5.73% over standard OPD) while cutting training trajectory length by 50.7%. The deeper lesson: supervision should concentrate at direction-changing points rather than being applied uniformly across every token.

This article analyzes Relay-OPD: Pass the Baton: Trajectory-Relayed On-Policy Distillation.

Paper: arXiv 2607.26057
Authors: Haolei Xu, Xiaowen Xu, Haiwen Hong, et al. (Zhejiang University / Alibaba Yuvion)
Code: https://github.com/zju-real/Relay-OPD
Project page: https://zju-real.github.io/Relay-OPD

Key points

The problem: prefix failure

On-Policy Distillation (OPD) lets a small student model generate its own reasoning trajectories, with a large teacher model providing token-level probability supervision. Because training and inference distributions match, OPD outperforms SFT. However, OPD has a structural flaw called prefix failure: once the student takes a wrong turn early in the chain of reasoning, everything afterward is a misled continuation. Token-level feedback on such trajectories is unreliable, and long failed trajectories waste training compute.

Existing fixes have drawbacks:

Fixed-length truncation (ESR, FastOPD): cuts at a fixed position regardless of where the failure actually occurs.
Offline rewriting (TRD): teacher rewrites the finished trajectory, but the rewrites carry visible "editing traces" and lose natural reasoning rhythm.
Token-level mixing (SKD): switches between teacher and student tokens based on distribution differences, which is too coarse a signal—distribution divergence doesn't mean directional error.

Key insight: teachers turn back, students keep going

The paper's central finding: on failing prefixes, the teacher tends to turn around while the student tends to push forward. At a fork in the reasoning, the teacher's top-1 token is typically a reflection word ("But", "Wait", "However"), while the student's top token is a continuation word ("So", "Therefore"). In one example (Figure 1a), the teacher assigns 74.4% to "But" while the student assigns 50.6% to "So" and nearly zero to "But".

This asymmetry requires no external labels or verifiers. The handoff criterion is:

\phi(h) = \mathbb{1}[a^T(h) \in \mathcal{R}] \cdot \mathbb{1}[\mathcal{K}_S(h) \cap \mathcal{R} = \emptyset]

where $$a^T(h)$$ is the teacher's top-1 token, $\mathcal{K}_S(h)$ is the student's top-K support set, and $\mathcal{R}$ is a predefined reflection-word set (Wait, But, However, and variants). The trigger fires only on directional disagreement, ensuring precise intervention.

The relay mechanism

When a handoff triggers:

1. The student generates the first leg of the trajectory. 2. The teacher takes over at the handoff point, producing a short corrective segment (usually a reflection token plus recovery reasoning). 3. Control returns to the student, which continues to the end. 4. The mixed relay trajectory is trained with standard OPD.

A relay budget (2–3 teacher interventions per trajectory) concentrates intervention at critical early positions and limits deviation from the student's own distribution. The entire rollout runs inside a speculative decoding engine, so the teacher intervenes by switching parameters rather than reloading models, keeping generation cost near single-model inference.

Results

Setup: teacher Qwen3-4B-Instruct-2507; students Qwen3-0.6B/1.7B (Non-Thinking); eight math benchmarks (AIME24/25/26, MATH, AMC23, Olympiad Bench, HMMT Feb26/Nov25).

| Method | Avg. accuracy | Trajectory length | |--------|--------------|-------------------| | Student baseline | 24.84 | — | | SFT | 33.20 | 4262 | | KD | 33.75 | 4262 | | GRPO | 34.42 | 2558 | | OPD | 41.23 | 4658 | | TRD | 30.69 | 2785 | | FastOPD | 45.47 | 2709 | | SKD | 42.35 | 4753 | | Relay-OPD | 46.96 | 2296 |

+5.73% over OPD, +1.49% over FastOPD; +7.29%/+7.19% on AIME25/AIME26; +3.01% for the 0.6B student.
Training trajectories shorten by 50.7% (4658 → 2296).
Compared to FastOPD, response lengths are 17.9%/14.2%/28.3% shorter on AIME25/AIME26/HMMT Feb26, with higher accuracy (+2.39%/+4.17%/+1.14%).

TRD underperforms standard OPD (30.69 vs 41.23 on 1.7B) due to rewrite artifacts; SKD gains only 1.12 points on 1.7B and loses 3.65 points on 0.6B, failing to break students' repetitive generation loops.

Conceptual positioning

Relay-OPD fits a recurring engineering principle: the granularity of optimization should match the granularity of the object being optimized. Token-level supervision (standard OPD) is too fine to see direction; trajectory-level rewriting (TRD) is too coarse and loses naturalness. Relay-OPD intervenes at the medium granularity of "direction-changing points"—both precise and natural.

Honest limitations

1. The reflection-word set $\mathcal{R}$ is hand-crafted; it works for math but may not transfer to code generation or open-ended dialogue. 2. Validation is limited to mathematical reasoning benchmarks. 3. The relay budget is a fixed hyperparameter with no adaptive adjustment by task difficulty. 4. The teacher-reflects/student-persists asymmetry was observed on Qwen3 models; other model families may behave differently.

Still, the mechanism-level finding stands on its own: the gap between strong and weak models lies not just in what they know, but in when they know to turn back. Supervision shouldn't be given only at the end, or uniformly at every step—it should be concentrated at direction-changing points.

FAQ

Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and deep learning.

Core takeaways? A relay-race metaphor; the prefix-failure problem in OPD; the teacher-student continuation asymmetry as a free handoff signal.

Is there open-source code? Yes: https://github.com/zju-real/Relay-OPD