The Pareto Paradox of One-Shot RLVR: Marginal Analysis of Data Scale from 1 to 1,200 Examples

Forum topic · 小凯 · 2026-05-11

Summary

A detailed analysis of the One-Shot RLVR paper (Wang et al., 2025, arXiv:2504.20571, NeurIPS 2025) showing that reinforcement learning with verifiable rewards (RLVR) achieves nearly identical performance whether trained on 1, 2, or 1,200 math problems. On MATH500, Qwen2.5-Math-1.5B improves from 36.0% to 73.6% with a single training example, matching 1,200-example training; average performance across six benchmarks rises from 17.6% to 35.7%. After decoupling format correction from genuine reasoning gains, both 1-shot and 1.2K-shot runs yield roughly the same non-format gain (~8.6%). Notable emergent behaviors include post-saturation generalization (test accuracy keeps improving long after training accuracy hits 100%), cross-domain generalization from a single geometry problem, and a standalone entropy-only loss that boosts MATH500 by 27.4% without any answer rewards. The effect is driven by policy gradient loss rather than grokking, and exploration—entropy—acts as the key catalyst. The article argues RLVR should shift from a data-scale race to a signal-quality race, with caveats: only math reasoning tested, no theory for post-saturation generalization, and sensitivity to the specific training problem chosen.

The Pareto Paradox of One-Shot RLVR: Marginal Analysis of Data Scale from 1 to 1,200 Examples

This post analyzes the paper "Reinforcement Learning for Reasoning in Large Language Models with One Training Example" (Wang et al., arXiv:2504.20571, NeurIPS 2025; code: https://github.com/ypwang61/One-Shot-RLVR).

Key points

Data scale is a pseudo-variable in RLVR: 1-shot training reaches parity with 1.2K-shot training. On MATH500 (Qwen2.5-Math-1.5B): baseline 36.0% → 1-shot 73.6% → 2-shot 74.8% → 1,200-shot 73.6%. Average across 6 benchmarks: 17.6% → 35.7% (1-shot) vs 35.9% (1.2K-shot).
The data-scaling curve resembles a step function rather than the logarithmic curve seen in supervised learning, suggesting RLVR "unlocks" existing capability rather than teaching new knowledge.
Format correction vs. real gains: a format-correction deduction protocol shows ~40-50% of the gain is format learning; the non-format reasoning gain is +8.6% for 1-shot and ~8.4% for 1.2K-shot — essentially identical. Both converge to the same policy basin.
Pure entropy loss alone lifts MATH500 by +27.4% (36.0% → 63.4%) with no answer rewards, and 1-shot + entropy reaches 73.6%. Exploration, not reward data, is the key catalyst.
Post-saturation generalization: training accuracy saturates (~100%) within ~100 steps, yet MATH500 test accuracy keeps improving until ~1,400 steps. Afterwards, outputs on the training problem degenerate into multilingual gibberish (though answers remain correct), while test outputs stay coherent.
Cross-domain generalization: training on a single geometry problem also improves algebra and number-theory accuracy — ruling out pure memorization, though transfer-of-reasoning vs. shared-format explanations remain undistinguished.
The effect stems from policy gradient loss, not grokking, and holds across models (Qwen2.5-Math-1.5B/7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B) and algorithms (GRPO, PPO).

Method and experimental design

The paper systematically compares 1-shot / 2-shot / 1.2K-shot RLVR, using one or two math problems from a DeepScaleR subset, evaluated on MATH500 plus five other math reasoning benchmarks. A format-correction protocol compares scores under strict vs. lenient formatting requirements: if gains persist under lenient formats, the improvement is genuine reasoning rather than oxed{}-style formatting.

Interpretation

Most of the 1,200 training problems likely lie in the model's "already mastered" zone (no signal) or "unreachable" zone (no positive reward). Only a few problems sit in the zone of proximal development, providing effective policy gradients — a single well-chosen problem suffices.

System-level implications

Data selection: shift from coverage/diversity metrics to signal strength — policy gradient variance and alignment between problem difficulty and model capability.
Stopping criteria: training-accuracy saturation is a poor stopping signal; test accuracy may keep improving for ~14x more steps.
Recommended practice: prioritize exploration mechanisms (higher entropy coefficient, β ≥ 0.01 suggested; randomized perturbations; temperature annealing) over data scale; adopt a phased data strategy (1-10 problems → validate; 10-100 → cross-validate; scale only if necessary).

Limitations

Only mathematical reasoning is tested; code, science, and commonsense domains are unverified.
Post-saturation generalization lacks a theoretical explanation; optimal training length cannot be predicted.
Cannot fully separate "genuine reasoning" from "sophisticated format matching."
Results are sensitive to which single problem is chosen; reproducibility is constrained.
Effects on 32B+ models and industrial-scale setups are untested.

Paper details

The Pareto Paradox of One-Shot RLVR: Marginal Analysis of Data Scale from 1 to 1,200 Examples

Summary

The Pareto Paradox of One-Shot RLVR: Marginal Analysis of Data Scale from 1 to 1,200 Examples

Key points

Method and experimental design

Interpretation

System-level implications

Limitations

Paper details

Tags