CriticOPD: Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Forum topic · 小凯 · 2026-10-16

Summary

On-policy distillation (OPD) trains a student model on its own rollouts using token-level supervision from a teacher, assuming the teacher can reach a correct answer from any prefix the student has written. The authors call this property teacher recoverability and show that it breaks down on incorrect rollouts: even though the teacher recognizes student errors, it tends to continue along the flawed path instead of correcting it. To address this, they propose CriticOPD, a method that uses the teacher's error recognition to restore recoverability. The teacher critiques an incorrect rollout, and the student rewrites its solution following that critique. Outputs that include the critique—treated as privileged information—are scored by the teacher, while the student is trained only on the revision without the critique. Experiments on two Qwen3 student models across six mathematical reasoning benchmarks show that CriticOPD consistently outperforms standard OPD, with the largest gains on competition-level math problems. The paper (arXiv:2610.20572) is by Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian, and was posted on 2026-10-14.

Paper Overview

Field: Machine Learning
Authors: Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian
Posted: 2026-10-14
arXiv: 2610.20572

Abstract

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves over OPD, with the largest gains on competition-level problems.

Key Points

Problem: OPD's token-level supervision assumes the teacher can recover a correct answer from any student prefix. This recoverability property fails on erroneous rollouts—the teacher detects the mistake but continues along it.
Method (CriticOPD):
The teacher critiques an incorrect rollout, identifying the error.
The student rewrites its solution following the critique.
The teacher scores outputs that include the critique (privileged information at scoring time only).
The student trains on the critique-free revision, so no privileged information leaks into the student's inference-time behavior.
Results: Evaluated on two Qwen3 student models across six mathematical reasoning benchmarks, CriticOPD consistently outperforms standard OPD, with the largest improvements on competition-style math problems.