Ocean in a Drop of Water: How One Training Example Unlocks 70% of AI Distillation Performance

Forum topic · 小凯 · 2026-09-05

Summary

A forum post on zhichai.net discusses the paper 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example' (arXiv:2609.04172). The paper reports a striking finding: in on-policy distillation (OPD), a single training example can drive hundreds of steps of improvement, recovering 70-87% of the performance gains from full-data training. On math reasoning, one example trained for 300 steps achieves 87% of full-dataset results. The authors attribute this to 'state coverage': stochastic rollouts from one sample cover 71.5% of the teacher's state space, and 16 semantically diverse examples reach 98.9%. They further show that the 'absorption rate'—how fast the student closes the gap to the teacher—is an intrinsic property of the optimization algorithm, not the data volume, leading to the conclusion that OPD is 'data-saturated but algorithm-hungry.' Content-light tests show OPD transfers general reasoning patterns rather than specific answers, contrasting with RLVR where one sample is nearly useless. The post outlines implications for post-training: curate a few state-diverse examples, focus on faster optimizers, and explore content-agnostic reasoning reinforcement.

Paper: Rethinking On-Policy Distillation of Large Language Models II: One Training Example Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo, et al. arXiv: 2609.04172

Key points

In on-policy distillation (OPD), 1 training example is enough to sustain hundreds of steps of improvement, recovering 70-87% of full-data performance gains (87% on math reasoning).
The result is stable regardless of the difficulty of the single training question, suggesting the specific content matters less than its structural role as a state generator.

How OPD works

Unlike supervised learning, OPD lets the student generate its own rollouts; the teacher provides token-level supervision on every step of the student's reasoning. Corrections target the student's own mistakes, making them highly relevant.

State coverage

Each reasoning step is a "state." Stochastic rollouts from 1 sample cover 71.5% of the state space of full-data training, mostly discovered in the first 100 steps.
With 16 semantically diverse samples, coverage reaches 98.9%—scaling from 16 to 17,000 examples yields minimal marginal gains.

The absorption rate bottleneck

The absorption rate — the fraction of student-teacher distance (in token probabilities) closed per step — is nearly identical with 1 sample (~78%) and full data (~84%). It is an intrinsic property of the optimizer, not the data. Hence the paper's conclusion:

> OPD is data-saturated and algorithm-hungry.

Content-independence

Training on nearly content-free inputs (empty chat templates, a bare <think/> prompt, off-domain chit-chat) still improves math benchmarks. OPD transfers general reasoning patterns (organizing thoughts, checking assumptions, avoiding skipped steps), not specific answers — like a writing coach critiquing a physics essay without knowing physics.

OPD vs. RLVR

| Feature | RLVR (e.g., GRPO) | OPD | |---|---|---| | Feedback | Sparse (right/wrong) | Dense (per-token) | | Signal persistence | Vanishes once correct | Always present | | 1-sample effectiveness | Nearly none | High |

In RLVR, once the student answers correctly the reward disappears; the teacher in OPD always has feedback, so the learning signal never runs dry.

Implications for post-training

1. Data curation: ~16 state-diverse examples may beat 17,000 random ones; measure state diversity, not quantity. 2. Algorithm work: faster optimizers, better losses, and update strategies may matter more than more data. 3. Multi-teacher distillation: ~16 representative examples per domain could drastically streamline building general models. 4. "Content-light" training: structured feedback alone might boost general reasoning ability without domain data.

Takeaway

Data quantity may be a red herring. What matters is (1) how many diverse states the data lets the model experience, and (2) how fast the algorithm can absorb feedback. The next breakthrough may come not from more data, but from methods that let models learn faster.

References

1. Fu, Z., He, B., Zuo, Y., et al. (2026). *Rethinking On-Policy Distillation of Large Language Models II: One Training Example*. arXiv:2609.04172. 2. Li, Y., et al. (2026). *Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe*. arXiv:2604.13016. 3. Agarwal, R., et al. (2024). On-policy distillation of language models: Learning from self-generated mistakes. *ICLR 2024*. 4. Wang, Y., et al. (2026). Reinforcement learning for reasoning in large language models with one training example. *NeurIPS 38*. 5. Gu, Y., et al. (2024). Minillm: Knowledge distillation of large language models. *ICLR 2024*.

*Post interpreted on 2026-09-06*