One Training Example Is Enough: The Data Paradox in On-Policy Distillation of LLMs

Paper Information

Title: Rethinking On-Policy Distillation of Large Language Models II: One Training Example Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo, et al. Categories: cs.AI, cs.CL

---

The Crazy Experiment

The standard recipe for distilling a large language model (teacher) into a smaller one (student) involves thousands of question-answer pairs and days of training. The authors of this paper asked an extreme question: what if you train on just a single query?

The experiment uses On-Policy Distillation (OPD): the student model generates its own responses (rollouts), and the teacher scores and guides them token-by-token. Instead of a large dataset, the researchers trained on one question only.

The Impossible Finding

> "One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families."

If full-data training scores 100 points, intuition says one sample might score 5–10. In reality, one-shot OPD reaches roughly 70–80 — like a friend studying a single problem 100 times and matching your score on a whole workbook of practice questions.

State Coverage

The paper introduces a key metric, state coverage:

> "State coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach."

Key results:

• A single query already reaches 71.5% state coverage, most of it within the first 100 training steps.
• 16 semantically diverse queries reach 98.9% and match full-data training.

The Data Paradox

The core conclusion:

> "OPD is therefore data-overfed but algorithm-starved."

Rollouts quickly expose broad supervision, but the student absorbs it very slowly:

> "Alignment slows at a similar pace whether OPD trains on one query or the whole dataset."

The bottleneck is not information volume but the student's limited capacity to digest supervision — analogous to a child learning to ride one bike well rather than trying 100 bikes for 5 minutes each.

Multi-Teacher and Cross-Domain Results

• Multi-teacher OPD (MOPD): 16 semantically diverse queries per domain match full-data multi-teacher distillation.
• Cross-domain generalization: even content-light templates (placeholder questions with little substance) and off-domain WildChat queries approach the real-query baseline.

> "Task content and induced state coverage can therefore come apart."

In other words: what matters is not *what* you train on, but *how much of the state space you explore*.

Implications

For LLM training: The traditional assumption "more data = better model" gives way to "better exploration = better model":

> "We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training."

For education and cognition: The finding echoes research on spaced repetition and elaborative encoding — a few items explored deeply from multiple angles beat broad, shallow coverage.

Conclusion

In OPD-based distillation, one well-chosen query can rival thousands of random ones. What matters is:

• 🎯 State coverage: does your data reach enough of the diverse state space?
• ⏱️ Step efficiency: can your algorithm effectively absorb supervision?
• 🧩 Semantic diversity: do your queries induce varied reasoning paths?

Depth beats breadth — at least in the world of on-policy distillation.

---

Reference: Fu, Z., He, B., Zuo, Y., et al. *Rethinking On-Policy Distillation of Large Language Models II: One Training Example*. arXiv:2609.01236.