Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation (GC-OPD) for Long-Context LLM Tasks

Forum topic · 小凯 · 2026-08-21

Summary

This arXiv paper (2608.19181) introduces GC-OPD, Group-Calibrated On-Policy Distillation, addressing a key weakness of on-policy distillation (OPD) in long-context tasks. The authors show that token-level teacher scores become progressively misaligned with task-specific verifier rewards as inputs grow longer, since teachers may favor locally plausible responses that omit distributed evidence or violate global constraints. GC-OPD normalizes verifier rewards and trajectory-level OPD scores within each rollout group, using the difference as a signed teacher-verifier disagreement residual. A Relative Advantage Credit Assignment (RACA) mechanism distributes this residual to tokens based on relative OPD advantage while preserving the original OPD signal. On five long-context benchmarks, GC-OPD raises the five-benchmark average of official Qwen3-4B checkpoints from 29.08 to 40.47 and Qwen3-8B from 35.12 to 44.65, outperforming vanilla OPD (39.31 and 43.56). Ablations confirm the signed residual and RACA are essential. Code is available on GitHub.

Overview

This paper (arXiv: 2608.19181, listed 2026-08-19) proposes Group-Calibrated On-Policy Distillation (GC-OPD), a method that incorporates task-specific verifier outcomes into on-policy distillation without discarding dense token-level teacher guidance.

Problem

On-policy distillation (OPD) trains a student model on its own responses using dense token-level guidance from a stronger teacher. However, in long-context tasks, token-level teacher support can favor locally plausible responses that:

omit evidence distributed across the input, or
violate global task constraints.

Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards reflecting partial success. The authors diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks: as inputs get longer, trajectory-level OPD scores become progressively less aligned with verifier rewards — a clear teacher-verifier disagreement.

Method: GC-OPD

Group calibration: Within each rollout group, verifier rewards and trajectory-level OPD scores are normalized separately, and their difference is used as a signed teacher-verifier disagreement residual.
RACA (Relative Advantage Credit Assignment): This trajectory-level residual is distributed to individual tokens according to their relative OPD advantage, while preserving the original OPD signal.

Results

On five long-context benchmarks, post-training with GC-OPD improves the five-benchmark averages of official checkpoints:

| Model | Baseline | GC-OPD | Vanilla OPD | |---|---|---|---| | Qwen3-4B | 29.08 | 40.47 | 39.31 | | Qwen3-8B | 35.12 | 44.65 | 43.56 |

Controlled ablations show the signed residual is more effective than adding extra OPD-derived terms or directly group-normalizing verifier rewards, and RACA outperforms uniform token allocation.

Code

Code: https://github.com/SolereZhang/GC-OPD

Authors

Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou (NLP).