One Training Example Is Enough: The Data Paradox in On-Policy Distillation of LLMs

Paper: *Rethinking On-Policy Distillation of Large Language Models II: One Training Example* Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo, et al. Categories: cs.AI, cs.CL

The Experiment

In standard large language model (LLM) distillation, practitioners prepare thousands of question-answer pairs and train a student model to imitate a teacher model's outputs. This paper asks an extreme question: what if you train on just one query?

The technique under study is On-Policy Distillation (OPD): the student model generates its own responses (rollouts), and the teacher model scores and guides these responses token by token. Typically this requires large training datasets—but the authors tested the data-minimal limit.

The Surprising Result

> "One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families."

If full-data training scores 100, intuition suggests one-shot training might reach 5–10. In practice, it reaches roughly 70–80.

Key Metric: State Coverage

The authors propose state coverage—the fraction of states visited by full-data OPD that a given query set's rollouts reach:

> "A single query already reaches 71.5%, most of it within the first 100 steps."

Scaling to 16 semantically distinct queries:

> "16 queries reach 98.9% and match full-data training."

The Data Paradox

The paper's core claim:

> "OPD is therefore data-overfed but algorithm-starved."

The student model quickly *sees* most of the supervision it needs through its rollouts, but absorbs that supervision very slowly. Evidence:

> "Alignment slows at a similar pace whether OPD trains on one query or the whole dataset."

The bottleneck is not the amount of information presented—it is the student's capacity to digest it, analogous to a child learning to ride a bike better with one bike and repeated practice than with 100 bikes ridden briefly each.

Multi-Teacher and Cross-Domain Findings

• Multi-teacher OPD (MOPD): "16 semantically diverse queries per domain match full-data MOPD."
• Cross-domain generalization: Even *content-light templates* (near-empty placeholder questions) and *off-domain WildChat queries* approach the real-query baseline:

> "Content-light templates and off-domain WildChat queries also approach the real-query baseline."

> "Task content and induced state coverage can therefore come apart."

In other words, what matters is not *what* you train on, but *how much of the state space you explore*.

Implications

For LLM training: The traditional assumption—more data equals better models—should be replaced by: better exploration equals better models. The authors state:

> "We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training."

Broader analogy: The finding resonates with spaced repetition and elaborative encoding in cognitive science—a small amount of content revisited deeply beats broad, shallow exposure. Deep beats wide.

The essentials for effective distillation:

• State coverage: does training expose a diverse state space?
• Step efficiency: can the algorithm absorb supervision effectively?
• Semantic diversity: do queries induce varied reasoning paths?

Reference

• Fu, Z., He, B., Zuo, Y., et al. *Rethinking On-Policy Distillation of Large Language Models II: One Training Example*. arXiv:2609.01236.