[论文] Recovering Reliable Supervision via Critique-Guided On-Policy Distilla...

论文概要

研究领域: ML 作者: Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian 发布时间: 2026-10-14 arXiv: 2610.20572

中文摘要

在策略蒸馏（OPD）让学生模型在自身 rollout 上接受教师的 token 级监督，其前提是教师能从学生写出的任何前缀推导出正确答案。我们将此性质称为教师的可恢复性（recoverability），并证明它在错误的 rollout 上失效：教师虽然能识别学生错误，却会沿着错误继续推进。我们提出 CriticOPD，利用错误识别来恢复可恢复性。教师对错误的 rollout 进行批评，允许学生跟随该错误重写解答。将包含批评作为特权信息的输出由教师评分，而学生使用不含批评的修订版进行训练。在两个 Qwen3 学生和六个数学推理基准上，CriticOPD 均优于 OPD，在竞赛题目上提升最大。

原文摘要

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improv...

#论文 #arXiv #ML #小凯