Rethinking On-Policy Distillation of LLMs II: One Query Is Enough to Recover Most Gains

This post shares an arXiv paper on on-policy distillation (OPD) of large language models.

• Field: NLP
• Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo
• arXiv: 2509.04282

Key points

• OPD combines student-generated rollouts with dense token-level supervision from a teacher; prior work focused on algorithmic behavior, leaving the role of training data unclear.
• The authors study the data-minimal limit: training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gains across task domains and model families.
• They introduce state coverage: the fraction of states visited by full-data OPD that a query set's rollouts reach. A single query reaches 71.5%, most within the first 100 steps.
• Adding semantically distinct queries raises both coverage and validation accuracy; 16 queries reach 98.9% coverage and match full-data training.
• Alignment speed between student and teacher slows at a similar pace whether training on one query or the full dataset — even a fixed state set takes hundreds of steps to absorb. Conclusion: OPD is data-surplus but algorithm-limited; rollouts expose supervision quickly while the student absorbs it ever more slowly.
• The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. Content-light templates and out-of-domain WildChat queries also approach real-query baselines, showing task content can be separated from the state coverage it induces.

The authors hope these findings shift future work toward OPD's step efficiency and the mechanisms behind its recent success in frontier post-training.

Original abstract (excerpt)

> On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches 71.5%, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy…

*Auto-collected 2026-09-07.*