Group-Calibrated On-Policy Distillation (GC-OPD): Combining Token-Level Teacher Guidance with Verifier Rewards

Forum topic · 小凯 · 2026-08-21

Summary

This paper introduces Group-Calibrated On-Policy Distillation (GC-OPD), a method for improving on-policy distillation (OPD) in long-context NLP tasks. The authors diagnose a key mismatch: on long inputs, trajectory-level OPD scores from a stronger teacher become progressively less aligned with task-specific verifier rewards, since token-level guidance can favor locally plausible responses that omit distributed evidence or violate global task constraints. GC-OPD normalizes verifier rewards and trajectory-level OPD scores within each rollout group, using their difference as a signed teacher-verifier disagreement residual. A Relative-Advantage Credit Assignment (RACA) mechanism distributes this residual to individual tokens based on their relative OPD advantage while preserving the original OPD signal. On five long-context benchmarks, GC-OPD post-training raises the five-benchmark average of Qwen3-4B from 29.08 to 40.47 and Qwen3-8B from 35.12 to 44.65, outperforming vanilla OPD (39.31 and 43.56). Ablations confirm the signed residual and RACA each contribute measurably. Code is available on GitHub.

Paper Overview

Field: NLP Authors: Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou arXiv: 2608.19181

Abstract

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success.

The authors diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement.

Method: GC-OPD

Motivated by this observation, the paper introduces Group-Calibrated On-Policy Distillation (GC-OPD):

Within each rollout group, verifier rewards and trajectory-level OPD scores are normalized separately.
Their difference serves as a signed teacher-verifier disagreement residual.
Relative-Advantage Credit Assignment (RACA) distributes this trajectory-level residual to individual tokens according to their relative OPD advantage, while preserving the original OPD signal.

Results

Across five long-context benchmarks, GC-OPD post-training improves the official Qwen3-4B and Qwen3-8B checkpoints' five-benchmark averages from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Under the same settings, vanilla OPD reaches 39.31 and 43.56.

Controlled ablations show that the signed residual is more effective than adding extra OPD-derived terms or directly group-normalizing verifier rewards, and RACA further outperforms uniform token allocation. These results indicate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance.