One Training Example Recovers 70% of On-Policy Distillation Gains

Forum topic · 小凯 · 2026-09-05

Summary

A forum post discusses the arXiv paper 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example' (arXiv:2609.04172). The study finds that in on-policy distillation (OPD), a single training example can drive hundreds of improvement steps, recovering 70-87% of the performance gains achieved with full datasets of ~17,000 examples. Using a 'state coverage' metric, the authors show that stochastic rollouts from one sample cover 71.5% of the full data's state space, while 16 diverse samples reach 98.9%. The paper argues OPD is 'data-oversaturated but algorithm-hungry': the bottleneck is the algorithm's absorption rate of teacher feedback, not data volume. Content-light experiments show gains even with empty templates, suggesting OPD teaches general reasoning patterns rather than answers. Compared to RLVR's sparse reward signals, OPD's dense token-level supervision never depletes. Implications include curating a few semantically diverse samples instead of massive datasets and prioritizing faster optimization algorithms in post-training.

Overview

This zhichai.net forum post explains the paper Rethinking On-Policy Distillation of Large Language Models II: One Training Example (Zixuan Fu, Bingxiang He, Yuxin Zuo, et al., arXiv:2609.04172).

Key points

What is OPD? In on-policy distillation, a student model (e.g., 1.5B) generates its own rollouts, and a strong teacher (e.g., DeepSeek-R1-70B) provides token-level feedback on each step — corrections are targeted at the student's own mistakes.
Core finding: Training on a single math problem for ~300 steps recovers 70–87% of the performance gains of full-dataset training. Reported recovery by domain: math reasoning 87%, code generation 73%, instruction following 68%, tool use 64%. Results are stable regardless of the seed problem's difficulty.
State coverage: One sample, via stochastic rollouts, covers 71.5% of the state space exposed by the full dataset; 16 semantically diverse samples reach 98.9%, making the marginal value of 17,000 samples tiny.
Absorption rate is the bottleneck: The fraction of student-teacher distance eliminated per step (~78% for 1 sample vs ~84% for full data) is nearly independent of data size — the algorithm, not the data, limits learning speed.
Content-light tests: Training on empty chat templates, a bare <think/> prompt, or non-math chitchat still improves math benchmarks, implying OPD transfers general reasoning patterns rather than content.
OPD vs RLVR: RLVR (e.g., GRPO) gives sparse, verifiable rewards that vanish once the student succeeds, so one example is nearly useless there. OPD's dense token-level supervision persists indefinitely.

Implications for post-training

Data curation: Prefer ~16 semantically diverse samples selected for state diversity over massive random datasets.
Algorithm research: New optimizers, loss functions, and update strategies that speed up the absorption rate matter more than more data.
Multi-teacher distillation: Each domain may need only a handful of representative samples, drastically simplifying post-training.
"Empty" training: Structured feedback without domain content could potentially boost baseline reasoning.

Conclusion

The paper's thesis: OPD is *data-oversaturated but algorithm-hungry* — rollouts rapidly expose a vast supervision space while the student absorbs it ever more slowly. The next breakthrough may come from making models learn faster, not from finding more data.

References

1. Fu, Z., He, B., Zuo, Y., et al. (2026). *Rethinking On-Policy Distillation of Large Language Models II: One Training Example*. arXiv:2609.04172. 2. Li, Y., et al. (2026). *Rethinking on-policy distillation of large language models*. arXiv:2604.13016. 3. Agarwal, R., et al. (2024). On-policy distillation of language models. *ICLR 2024*. 4. Wang, Y., et al. (2026). RL for reasoning with one training example. *NeurIPS 38*. 5. Gu, Y., et al. (2024). MiniLLM: Knowledge distillation of large language models. *ICLR 2024*.