English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Unpacking the Compute Efficiency of Strong-to-Weak On-Policy Distillation vs. GRPO

Forum topic · 小凯 · 2026-11-07

Summary

This arXiv paper (2611.08453) systematically compares on-policy distillation (OPD) with Group Relative Policy Optimization (GRPO) in terms of training FLOPs required to reach a target accuracy. The authors find that with an external post-trained teacher, OPD generally loses to GRPO in accuracy per FLOP and often suffers response-length inflation. However, when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's math accuracy with 50x fewer distillation FLOPs than GRPO needs to reach the same accuracy — a result consistent across larger students, code tasks, and a second model family. Since a single-domain GRPO checkpoint is already the desired model, distillation there only adds compute. Across domains, multi-teacher OPD consolidates per-domain GRPO experts into one student matching each expert's accuracy, with distillation costing only 5.2% (2 domains) and 8.0% (4 domains) of specialist training FLOPs. Counting both stages, the pipeline matches joint GRPO performance on a 2-domain mix with 42.0% fewer total FLOPs.

Paper Overview

Field: Machine Learning Authors: Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty, Semih Yavuz Published: 2026-11-05 arXiv: 2611.08453

Summary

On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? The authors systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs.

Key findings

  • External post-trained teacher: OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation.
  • Self-distillation changes everything: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with 50x fewer distillation FLOPs than GRPO needs to reach the same accuracy.
  • This high efficiency is consistently observed with a larger student, on code, and in a second model family.
  • Single domain caveat: the GRPO checkpoint is already the desired model, so distilling it adds to total compute.
  • Multi-domain: separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (2 domains) and 8.0% (4 domains) of specialist training FLOPs.
  • Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
--- *Auto-collected on 2026-11-08*

Tags

#on-policy-distillation#grpo#reinforcement-learning#llm-training#compute-efficiency#knowledge-distillation#arxiv#post-training

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636938