One Training Example Is Enough: On-Policy Distillation Recovers 70-87% of Full-Data Gains in LLMs

Forum topic · 小凯 · 2026-09-05

Summary

A forum post on zhichai.net reviews the paper 'Rethinking On-Policy Distillation of Large Language Models II: One Training Example' (arXiv:2609.04172), which reports a counterintuitive finding: in on-policy distillation (OPD), a single training example can drive hundreds of optimization steps and recover 70-87% of the performance gains achieved with the full dataset. On math reasoning, one example restores 87% of full-data accuracy; code generation, instruction following, and tool use show similar effects. The mechanism is explained via 'state coverage': stochastic rollouts from one sample cover 71.5% of the teacher's supervision state space, and 16 semantically diverse samples reach 98.9% coverage. The post argues OPD is 'data-saturated but algorithm-hungry'—absorption rate, the speed at which the student closes the gap to the teacher, is an intrinsic property of the optimizer rather than the data volume. Dense token-level supervision from the teacher means learning signals never run out, unlike sparse-reward RLVR. Content-light tests show even near-empty prompts improve math ability, suggesting OPD transfers general reasoning patterns rather than content. Implications include prioritizing state diversity over dataset size and developing faster-absorbing optimizers.

One Training Example Is Enough: On-Policy Distillation Recovers 70-87% of Full-Data Gains

Paper: Rethinking On-Policy Distillation of Large Language Models II: One Training Example Authors: Zixuan Fu, Bingxiang He, Yuxin Zuo, et al. arXiv: 2609.04172

This post is an editorial walkthrough of the paper, presenting its counterintuitive finding with analogies and analysis.

The Counterintuitive Result

The paper reports that in on-policy distillation (OPD), a single training example is sufficient to sustain improvement over hundreds of steps, recovering 70-87% of the performance gains from full-dataset training. The authors summarize:

> OPD is "data-oversaturated and algorithm-hungry."

What Is On-Policy Distillation (OPD)?

OPD transfers a teacher model's (e.g., DeepSeek-R1-70B) reasoning into a small student model (e.g., 1.5B):

1. The student generates its own solution (rollout) to a problem. 2. The teacher provides token-level supervision on every step of the student's reasoning—not just the final answer. 3. The student updates based on this feedback.

The key is "on-policy": corrections target the student's *own* mistakes, making the feedback maximally relevant.

The One-Sample Miracle

Training a student on one math problem for 300 steps yields:

| Domain | % of full-data gain recovered | Full-data accuracy | |---|---|---| | Math reasoning | 87% | 72.1% | | Code generation | 73% | - | | Instruction following | 68% | - | | Tool use | 64% | - |

One sample trained for 300 steps matches 87% of what 17,000 samples achieve in the same number of steps. The result holds regardless of whether the initial problem is easy, medium, or hard for the student—the specific content matters less than its structural role as a state generator.

State Coverage: The Mechanism

Each reasoning step is a "state" in the model's inference trajectory.
Stochastic rollouts from one sample cover 71.5% of the state space reached by the full dataset.
Most new states are discovered within the first ~100 training steps.
With 16 semantically diverse samples, coverage reaches 98.9%—increasing to 17,000 samples adds almost nothing.

Absorption Rate: The Real Bottleneck

The authors track the absorption rate: the fraction of student-teacher distance (in token probabilities) eliminated per step, v_t = (d_t - d_{t+1}) / d_t.

| Data | Absorption rate at step 300 | Cumulative distance removed | |---|---|---| | 1 sample | ~78% | ~78% | | Full data | ~84% | ~84% |

Absorption is an intrinsic property of the optimizer, not the data volume. The bottleneck of OPD is the algorithm (e.g., AdamW), which absorbs teacher signal too slowly—not the data.

Content-Invariance

In a "content-light test," the authors trained models on nearly content-free inputs (an empty chat template, a bare <think/> prompt, off-topic chitchat queries). Models still improved on math benchmarks—because OPD teaches general reasoning patterns (how to organize thoughts, when to check assumptions), not specific answers. The teacher's token-level corrections remain informative even on nonsensical student outputs.

OPD vs. RLVR

| Feature | RLVR (e.g., GRPO) | OPD | |---|---|---| | Feedback | Sparse (right/wrong) | Dense (per-token) | | Signal persistence | Vanishes once solved | Always present | | 1-sample efficacy | Nearly none | High |

In RLVR, once the student answers correctly, the reward disappears. In OPD, the teacher always has feedback, so dense supervision never runs dry—explaining why one sample works in OPD but not in RLVR.

Implications for Post-Training

1. Data curation: pick ~16 semantically diverse samples instead of curating millions; optimize for state diversity, not quantity. 2. Algorithm research: since data is not the bottleneck, faster-absorbing optimizers, losses, and update strategies matter most. 3. Multi-teacher distillation: each domain may need only ~16 representative samples. 4. "Empty" training: content-light results hint at domain-agnostic reasoning-strengthening methods based purely on structured feedback.

Conclusion

Data quantity may be a red herring. What matters is (1) how many diverse states the data lets the model visit, and (2) how quickly the algorithm absorbs feedback. The next breakthrough in post-training may come not from more data, but from methods that let models learn faster.

References

1. Fu, Z., He, B., Zuo, Y., et al. (2026). *Rethinking On-Policy Distillation of Large Language Models II: One Training Example*. arXiv:2609.04172. 2. Li, Y., et al. (2026). *Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe*. arXiv:2604.13016. 3. Agarwal, R., et al. (2024). On-policy distillation of language models: Learning from self-generated mistakes. *ICLR 2024*. 4. Wang, Y., et al. (2026). Reinforcement learning for reasoning in large language models with one training example. *NeurIPS 38*. 5. Gu, Y., et al. (2024). MiniLLM: Knowledge distillation of large language models. *ICLR 2024*.

*Published on zhichai.net; original commentary dated 2026-09-06.*