Rethinking On-Policy Distillation of LLMs II: One Query Is Enough to Recover Most Gains

Forum topic · 小凯 · 2026-09-07

Summary

This arXiv paper (2509.04282) by Zixuan Fu, Bingxiang He, and Yuxin Zuo investigates the role of training data in on-policy distillation (OPD) of large language models by training on a data-minimal limit: a single query. One-shot OPD continues improving for hundreds of steps and recovers most of the gains of full-data OPD across task domains and model families. The authors explain this via state coverage—the fraction of full-data OPD states reachable by a query set's rollouts. A single query reaches 71.5% coverage, mostly within the first 100 steps; 16 semantically distinct queries reach 98.9% and match full-data training. Alignment speed slows at a similar rate regardless of data volume, showing OPD is data-redundant but algorithm-limited: rollouts expose broad supervision quickly, while the student absorbs it increasingly slowly. Findings extend to multi-teacher OPD and lightweight out-of-domain templates, suggesting task content can be decoupled from induced state coverage.

Paper Overview

Field: NLP Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo Published: 2026-09-06 arXiv: 2509.04282

Abstract

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. The authors examine this role at the data-minimal limit by training on a single query.

Key findings

One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families.
State coverage — the fraction of the states full-data OPD visits that a query set's rollouts reach — explains the result. A single query already reaches 71.5%, most of it within the first 100 steps.
Adding semantically distinct queries raises coverage and validation accuracy until 16 queries reach 98.9% and match full-data training.
Alignment speed slows at a similar pace whether OPD trains on a single query or an entire dataset — even a fixed state set takes hundreds of steps to absorb. OPD is therefore data-redundant but algorithm-limited: its rollouts quickly expose broad supervision, while the student absorbs it increasingly slowly.
The state coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD.
As a stress test, content-light templates and out-of-domain WildChat queries also approach real-query baselines — task content and induced state coverage can be decoupled.

The authors hope these findings steer future work toward OPD's step efficiency and re-examination of its data and the mechanisms behind its recent success in frontier post-training.