[论文] Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO

论文概要

研究领域: ML 作者: Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty, Semih Yavuz 发布时间: 2026-11-05 arXiv: 2611.08453

中文摘要

在策略蒸馏（OPD）让学生在冻结教师的逐 token 监督下，基于自己生成的 rollout 进行训练。这种密集信号能否比强化学习用更少的训练 FLOPs 达到目标精度？我们系统地对比了 vanilla reverse-KL OPD 与 GRPO，横跨任务难度、学生/教师规模、领域和优化选择，以 held-out 精度对训练 FLOPs 做度量。使用外部后训练教师时，OPD 在等设置下每 FLOP 的精度普遍不如 GRPO——即使教师本身也用 GRPO 进一步训练过——且经常出现回复长度膨胀。但换一种教师就完全不同：当教师就是学生自己的 GRPO checkpoint 时，OPD 仅用 GRPO 达到同等精度所需蒸馏 FLOPs 的 1/50，就能在数学上恢复该 checkpoint 的精度。在更大规模学生、代码任务和第二个模型家族上，我们都一致观察到了这种高效率。但在单一领域内，GRPO checkpoint 本身就是目标模型，再做蒸馏反而增加总算力。跨领域时，各自的 GRPO checkpoint 能力不同，没有一个能通吃。多教师 OPD 将各领域 GRPO 专家整合到单一学生中，使其匹配每个专家的精度，蒸馏阶段仅消耗专家训练 FLOPs 的 5.2%（2 个领域）和 8.0%（4 个领域）。总算账，该流水线在 2 领域混合上达到联合 GRPO 的性能，总 FLOPs 减少 42.0%。

原文摘要

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with 50x fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.

---

*自动采集于 2026-11-08*

#论文 #arXiv #ML #小凯