One Number That Keeps Engineers Up at Night
In April 2025, researchers from Microsoft Research and the University of Washington published a paper with an unremarkable-sounding title:
> Reinforcement Learning for Reasoning in Large Language Models with One Training Example
But its first result is enough to make anyone who has run RL training jump out of their chair. They took Qwen2.5-Math-1.5B, ran RL training on a single problem, and evaluated on MATH500 (500 competition-level math problems):
| Training data | MATH500 accuracy | Avg. of 6 benchmarks | |:---:|:---:|:---:| | 0 (baseline) | 36.0% | 17.6% | | 1 problem | 73.6% | 35.7% | | 2 problems | 74.8% | 36.6% | | 1,200 problems | 73.6% | 35.9% |
> MATH500: a benchmark of 500 competition-level math problems spanning algebra to geometry, built by Hendrycks et al. > > DeepScaleR subset (DSR-sub): a curated set of 1,200 math problems widely used as a standard RL training dataset.
One problem doubles accuracy. Two problems beat a thousand.
This is not a typo. The result held across multiple models (1.5B, 7B, Llama3.2, DeepSeek-R1-Distill) and algorithms (GRPO, PPO): in RLVR (Reinforcement Learning with Verifiable Rewards), data scale barely matters.
The Industry's Collective Illusion
In the two-year arms race to reproduce DeepSeek-R1, an unwritten rule took hold: more data is better. SimpleRL used 8.5K problems, DeepScaleR 40K, PRIME 150K. Teams raced to collect and clean math datasets.
This paper slaps that assumption. A 1,200-problem dataset carefully curated over months performs the same as a randomly picked single problem.
Even more striking: much of the gain doesn't come from the math content at all—it comes from the model learning an output format during RL.
> Format correction: baseline models often output answers in non-standard formats (e.g., missing the \boxed{} marker) that automated graders cannot parse. Early RL training quickly teaches the model to write answers in the grader's preferred format. This pure-format gain accounts for roughly 5–10 points of MATH500 improvement.
After carefully subtracting format correction, one-shot RLVR still delivers an 8.6-point true reasoning gain—essentially identical to the true gain from 1.2K-problem training. The model didn't "learn math" from 1,200 problems; it "learned how to answer" from one.
What Does That One Problem Teach?
The key clue: the chosen problem was already easy for the base model. The baseline Qwen2.5-Math-1.5B already solves 36% of MATH500—it has the knowledge but lacks stable reasoning strategies and output habits.
RL doesn't teach new knowledge; it consolidates existing scattered capabilities into stable inference habits. It's like someone who can walk but stumbles—you don't need a thousand different roads, just enough repetition on one road until the gait corrects itself. Then every other road gets easier too.
The paper's strangest experiment: using only entropy loss, with no answer rewards at all, improved MATH500 accuracy by 27.4%. Entropy loss merely encourages the model to explore diverse output forms—without ever telling it what's "correct," the model plays its way into better reasoning formats.
Getting Smarter or Getting Crafty?
The paper records an unsettling phenomenon: after ~1,400 steps, the model's outputs on the single training problem degenerate into multilingual gibberish—mixed Chinese, English, math symbols, meaningless tokens—yet the answer stays correct.
Stranger still: training accuracy saturated at 100%, but test accuracy kept rising. The authors call this Post-Saturation Generalization—by conventional ML theory, training should have stopped. Instead, over-optimizing on one problem apparently uncovered transferable reasoning patterns.
One encouraging detail: the model's self-reflection frequency on unseen test problems increased noticeably (phrases like "Wait...", "Let me verify..."). A model merely gaming the grader has no reason to reflect on problems nobody is training it on. This suggests something real was learned, not just performed.
But—this is not fully certain.
The Most Uncomfortable Implication
If one problem suffices to teach reasoning, the industry's two-year data-hoarding spree was a collective superstition. Those 1,200 / 40K / 150K numbers weren't engineering necessities—they were psychological placebos making teams feel thorough, papers look solid, and reviewers feel reassured.
In the RLVR framework, data scale may be a pseudo-variable. What matters is whether the model hits the tipping point—the exploration-driven transition past a potential barrier in output format or strategy—into a more stable basin of reasoning behavior.
This raises a deeper question: how do we know the model "truly understands math" rather than "found a trick that scores well on the test set"? Cross-domain generalization (training on one geometry problem lifting algebra and number theory too) offers partial comfort, but doesn't rule out all test problems sharing implicit structure—similar notation, phrasing, grading conventions.
The most uncomfortable conclusion: we may never distinguish "true reasoning" from "sophisticated pattern matching"—and for most practical purposes, the distinction may not matter. If the model is consistently right on tests, who cares whether it reasons "inside"?
The Bet
My judgment: the "big data requirement" in RLVR is a hallucination amplified by industry inertia. With the right exploration mechanisms, a single problem can trigger the emergence of reasoning. Teams still competing on "how many K of data I used" are making the same cognitive error as the teams that once competed on parameter counts.
My identified enemy: the industry subtext that "data scale = research seriousness = model quality." My price if wrong: if one-shot RLVR is a coincidence of specific models, domains, and algorithms—failing to transfer to code, science, or multimodal reasoning—then my judgment is small-sample survivorship bias. I'll take that reputation.
The scariest possibility: we may be training super test-takers—models that don't "understand" math but outperform any human at scoring high on specific exam formats. Even so, I'm betting on "less is more," because real learning isn't about how many problems you grind—it's about whether you find the moment of insight.
> Next time you hear about an RL project using 100K training samples, don't ask "that many?"—ask "how much worse would one sample be?"
Paper Details (Verified)
| Item | Detail | |:---:|:---| | Title | Reinforcement Learning for Reasoning in Large Language Models with One Training Example | | Authors | Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen | | Institutions | University of Washington, USC, Microsoft, UC Santa Cruz, Georgia Tech | | arXiv | 2504.20571 (Apr 29, 2025), NeurIPS 2025 | | Code | https://github.com/ypwang61/One-Shot-RLVR | | Models verified | Qwen2.5-Math-1.5B / 7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B | | Algorithms | GRPO, PPO | | Key results | 1-shot: MATH500 36.0%→73.6% (+8.6% non-format gain), 6-benchmark avg 17.6%→35.7%; 2-shot: MATH500 74.8%, avg 36.6%; matches 1.2K-problem training | | Key findings | Post-Saturation Generalization; Cross-Domain Generalization; +27.4% from pure entropy loss; effect stems from policy gradient loss, not grokking |