Rethinking On-Policy Distillation of LLMs II: One Trajectory May Be Enough

Forum topic · 小凯 · 2026-09-07

Summary

This paper (arXiv:2509.04282) examines the role of training data in on-policy distillation (OPD), where a student LLM learns from its own rollouts with dense token-level supervision from a teacher. By training at the data-minimal limit—on a single query—the authors show that one-shot OPD keeps improving for hundreds of steps and recovers most of the gains of full-data OPD across task domains and model families. They introduce a metric called state coverage: a single query already reaches 71.5% of the states visited by full-data OPD, mostly within the first 100 steps. Adding semantically distinct queries increases coverage and validation accuracy until 16 queries reach 98.9% and match full-data training. Crucially, alignment with the teacher slows at a similar pace regardless of dataset size, indicating OPD is data-abundant but algorithm-limited: rollouts expose broad supervision quickly, but the student absorbs it ever more slowly. Findings extend to multi-teacher OPD, and content-light templates or out-of-domain WildChat queries nearly match real-query baselines, showing task content can be decoupled from state coverage.

Paper Overview

Field: NLP Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo arXiv: 2509.04282

Key Points

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Prior work has studied its algorithmic behavior, but the role of training data remained unclear.
The authors probe this at the data-minimal limit: training OPD on a single query. One-shot OPD continues to improve for hundreds of steps and recovers most of full-data OPD's gains across task domains and model families.
State coverage is introduced as an explanatory metric: the fraction of states visited by full-data OPD that a query set's rollouts reach.
A single query already reaches 71.5%, with most of that achieved within the first 100 training steps.
Adding semantically distinct queries raises both coverage and validation accuracy; 16 queries reach 98.9% and match full-data training.
Alignment speed is the bottleneck: the student's alignment with the teacher slows at a similar rate whether training on one query or the full dataset—even a fixed set of states takes hundreds of steps to absorb. OPD is therefore *data-abundant but algorithm-limited*: rollouts expose broad supervision quickly, but the student absorbs it increasingly slowly.
The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD.
Stress tests: content-light templates and out-of-domain WildChat queries nearly match real-query baselines—task content can be decoupled from the state coverage it induces.

Implications

The authors hope these findings shift future work toward the step efficiency of OPD and prompt a re-examination of its data requirements and the mechanisms behind its recent success in frontier post-training.

---

*Auto-collected on 2026-09-07.*