Overview
This post analyzes "Reinforcement Learning for Reasoning in Large Language Models with One Training Example" (Wang et al., 2025, arXiv:2504.20571, NeurIPS 2025). In the wave of open-source efforts reproducing DeepSeek-R1, RLVR dataset sizes have ballooned from 8K to 150K examples — yet no one has examined the marginal returns of scale. This paper does, with striking results.
Key points
- Data scale is nearly a pseudo-variable: 1-shot RLVR lifts MATH500 from 36.0% → 73.6% (6-benchmark average 17.6% → 35.7%). 2-shot reaches 74.8%, and 1,200 examples reach 73.6% — essentially zero marginal utility beyond the first example.
- The curve is a step function, not logarithmic: performance ≈ c·1[N ≥ 1] + d, suggesting RLVR unlocks pre-existing capabilities rather than teaching new knowledge.
- Format correction vs. real gains: using a lenient-format evaluation protocol, roughly 40–50% of the gain is format correction (e.g., learning
\boxed{}), but the non-format gain of +8.6% is nearly identical for 1-shot and 1.2K-shot — both converge to the same policy basin. - Pure entropy loss alone gives +27.4%: without any answer rewards, maximizing policy entropy lifts MATH500 to 63.4%, implying self-organized exploration — models just need to be encouraged to "try more things," not told what's correct.
- Post-Saturation Generalization: training accuracy on the single problem saturates at 100% within ~100 steps, yet test accuracy keeps climbing until ~1,400 steps. Afterward, outputs on the training problem degenerate into multilingual gibberish (while remaining correct), but test outputs stay clean and readable.
- Cross-domain generalization: training on one geometry problem also improves algebra and number theory scores, ruling out pure memorization (though transfer vs. shared-format explanations remain unresolved).
- Mechanism: the effect is driven by policy gradient loss, not grokking; exploration (entropy) is the critical catalyst.
- Models: Qwen2.5-Math-1.5B/7B, Llama-3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B
- Algorithms: GRPO, PPO
- Training data: one or two math problems from the DeepScaleR subset
- Evaluation: MATH500 plus 5 other math benchmarks
- Code: https://github.com/ypwang61/One-Shot-RLVR
- Math-only validation; code, science, and commonsense reasoning untested
- No theoretical explanation for Post-Saturation Generalization or the inverted overfitting-generalization relationship
- Cannot fully separate "true reasoning" from sophisticated format matching
- Results are sensitive to which single problem is chosen; 32B+ models untested
- Title: Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Authors: Yiping Wang et al. (University of Washington, USC, Microsoft, UC Santa Cruz, Georgia Tech)
- arXiv: 2504.20571, April 29, 2025, NeurIPS 2025
- Code: https://github.com/ypwang61/One-Shot-RLVR
Experimental setup
Implications
1. Optimize exploration, not scale: use higher entropy coefficients (β ≥ 0.01 suggested), heavy-tailed sampling, and stochastic perturbations during training. 2. Redefine data efficiency: move beyond Performance/N toward metrics accounting for generalization, training steps, and compute cost. 3. Adopt staged data strategies: validate with 1–10 examples first, cross-check with 10–100, and scale up only when necessary. 4. Rethink stopping criteria: training-accuracy saturation is a poor signal — test gains may continue 14× longer.