One Training Example Is Enough: When RL Discovers That Learning to Reason Doesn't Need Big Data

1. A Number That Keeps Engineers Awake 🤯

In April 2025, researchers from Microsoft Research and the University of Washington published a paper with a plain, unassuming title:

> "Reinforcement Learning for Reasoning in Large Language Models with One Training Example"

But the first number in the paper is enough to make anyone who has run RL training jump out of their chair.

They took Qwen2.5-Math-1.5B (a base math model) and ran RL training on a single problem.

Then they evaluated on MATH500 (500 competition-level math problems).

The result:

| Training data | MATH500 accuracy | 6-benchmark average | |:---:|:---:|:---:| | 0 (baseline) | 36.0% | 17.6% | | 1 problem | 73.6% | 35.7% | | 2 problems | 74.8% | 36.6% | | 1,200 problems | 73.6% | 35.9% |

> MATH500: A benchmark of 500 competition-level math problems built by Hendrycks et al., spanning algebra to geometry; the standard dataset for testing LLM math reasoning. > > DeepScaleR subset (DSR-sub): 1,200 curated math problems from the DeepScaleR project, widely used as a standard RL training dataset.

One problem doubles accuracy. Two problems beat a thousand.

This is not a typo. The paper validated the result across multiple models (1.5B, 7B, Llama3.2, DeepSeek-R1-Distill) and multiple algorithms (GRPO, PPO). The conclusion held: in RLVR (Reinforcement Learning with Verifiable Rewards), data scale barely matters.

2. The Industry's Collective Illusion: We Thought We Were "Teaching Math," but We Were "Buying Lottery Tickets" 🎰

> RLVR (Reinforcement Learning with Verifiable Rewards): An RL paradigm where the model solves problems and receives rewards based on answer correctness. For math, the reward is binary: +1 for correct, 0 or negative for wrong. Since rewards are automatically verifiable (no human annotation needed), it's called "verifiable rewards."

Over the past two years, in the open-source arms race to reproduce DeepSeek-R1, there has been an unwritten rule: more data is better.

SimpleRL used 8.5K problems. DeepScaleR used 40K. PRIME used 150K. Everyone scrambled to collect, clean, and annotate math datasets, as if whoever had the most data would have the strongest model.

But Yiping Wang and colleagues' paper is a slap in the face to that assumption.

They found something astonishing: the 1,200-problem dataset you spent months curating performs exactly the same as one randomly sampled problem.

Even more astonishing: most of the improvement doesn't even come from "the content of the math problems" — it comes from the model learning some output format during RL training.

> Format Correction: A common behavior change early in RL training. The base model may output answers in non-standard formats (e.g., missing the \boxed{} marker), causing the automatic grader to fail parsing. RL training quickly teaches the model to "write answers in the grader's preferred format." This purely format-level improvement accounts for roughly 5-10 points of MATH500 accuracy growth.

The paper carefully subtracted the format-correction contribution. Even without it, 1-shot RLVR's "genuine reasoning gain" is still 8.6 points — nearly identical to the genuine gain from 1.2K-problem training.

What does this mean? The model didn't "learn math knowledge" from 1,200 problems. It "learned how to answer" from one.

3. What Did That One Problem Actually Teach? 🔑

The paper doesn't over-interpret, but it offers a key clue: the "magic problem" was already easy for the base model.

> Base Model: The original pretrained model without RL training. Qwen2.5-Math-1.5B's baseline already reaches 36% on MATH500, meaning it possesses some math knowledge — it just lacks a stable reasoning strategy and output format.

In other words, the model could already solve the problem (with high probability). RL's role is not "teaching new knowledge" but "solidifying existing, scattered capabilities into stable reasoning habits."

It's like a person who can already walk but stumbles. You don't need to take them down a thousand different roads — you just need them to walk one road repeatedly until their gait is right. Then they'll be surprised to find they walk steadily on other roads too.

> Policy Gradient: The core RL method that directly optimizes the policy (the model). The paper shows that 1-shot RLVR's effect mainly comes from the policy gradient loss, not other mechanisms (such as value function estimation or model compression).

The paper also ran a stranger experiment: using only entropy loss — no answer rewards at all — still improved MATH500 accuracy by 27.4%.

> Entropy Loss: A technique in RL that encourages exploration. It penalizes overly "certain" output distributions (always giving one answer) and rewards staying "open" (considering multiple possibilities). Pure entropy loss never tells the model "what's correct" — only "don't conclude too early."

This 27.4% improvement comes entirely from the model being encouraged to explore more possible output forms — it "played" its way into a better reasoning format without any supervision from correct answers.

4. Is the Model Getting Smarter, or Just Getting "Sly"? 🦊

The paper records a disturbing phenomenon: the model's outputs on the training problem gradually became multilingual gibberish.

Around 1,400 training steps in, the model's answers to the single training problem started mixing Chinese and English, math symbols, and meaningless tokens — but eerily, the answer was still correct.

> Overfitting: When a model memorizes training data so thoroughly that it produces non-generalizable, distorted output patterns on training samples. Traditionally, overfitting means test performance degrades; but in 1-shot RLVR, even as the model goes "crazy" on the training problem, test performance stays strong.

Stranger still, the test-time behavior:

• Training accuracy saturated at 100% long ago
• But test accuracy kept rising

> A counterintuitive phenomenon. By traditional ML theory, once the model achieves perfection on training data (100% accuracy), training should stop to avoid overfitting. But in 1-shot RLVR, test performance keeps improving even as training continues. This suggests that during "over-optimization" on the training problem, the model discovers some transferable reasoning pattern.

An analogy: a student practices one problem a thousand times, until the answer itself becomes muscle memory. But along the way, he inadvertently invents a set of problem-solving "hand gestures" — gestures that make him faster and more accurate on other problems. He wasn't practicing the problem; he was practicing the technique.

The question: is this "technique" genuine reasoning, or gaming the grader?

The paper doesn't give a final answer. But I noticed one detail: the model's self-reflection frequency on test problems increased.

> Self-Reflection: The model's behavior of actively questioning itself and checking steps during inference. In chain-of-thought, this appears as phrases like "Wait...", "Let me verify...", "Actually...". The paper found that after 1-shot RLVR, the model uses such phrases significantly more often on test problems — even though the training problem itself didn't require reflection.

If the model were just "pretending to be smart" to fool the grader, it wouldn't need to self-reflect on unsupervised test problems. That detail makes me lean toward believing: something was genuinely "learned," not merely "performed."

But — I'm not entirely sure.

5. The Most Uncomfortable Inference 😰

At this point, I must share an inference that sent a chill down my spine:

If one problem is enough to teach a model to reason, then the past two years of industry-wide dataset hoarding is essentially a collective superstition.

1,200 problems, 40K problems, 150K problems — these numbers aren't engineering necessities; they're psychological placebos. They make teams feel they've "done enough," make papers look "solid enough," make reviewers feel "confident enough."

But the truth may be: under the RLVR framework, data scale is a pseudo-variable. What matters isn't "how many problems you fed," but "whether the model hit that critical format/strategy tipping point during exploration."

> Tipping Point: In complex systems, small initial changes get amplified through positive feedback loops, causing a qualitative state change. In 1-shot RLVR, entropy loss or a single problem's reward signal may push the model over some "reasoning format" potential barrier, after which it enters a more stable, more efficient policy basin.

This raises a deeper question: if one problem suffices, how do we know the model "truly understands math," rather than having just "found a trick to score high on the test set"?

The paper's cross-domain generalization offers partial comfort: training on one geometry problem also improved algebra and number theory. But this doesn't rule out the possibility that all test problems share some implicit structure — similar notation, similar phrasing, similar grading criteria.

> Cross-Domain Generalization: Performance gains on one domain (e.g., algebra) after training on another (e.g., geometry). This is usually taken as evidence of "genuine learning," since it suggests transferable reasoning rather than domain-specific memorization. But generalization could also stem from shared surface-level format features across domains, not deep understanding.

The most uncomfortable inference: we may never be able to distinguish "genuine reasoning" from "sophisticated pattern matching" — and worse, for most practical applications, the distinction may not matter at all. If the model is consistently correct on the test set, who cares whether it's "really" reasoning inside?

But I care. At least in this post, I care.

6. The Bet 🎯

Alright, I'll place my bet.

My judgment: the "big data requirement" in RLVR is an illusion amplified by industry inertia. With the right exploration mechanisms, a single problem is enough to trigger the emergence of a model's reasoning capability. Teams still competing over "how many K of data I used" are making the same cognitive error as teams that once competed over "how many B of parameters I used."

The enemy I'm identifying is this industry subtext: "data scale = research seriousness = model quality." It's not real, it's popular, and it makes annotation companies and cloud vendors laugh all the way to the bank.

The price I'm paying: if 1-shot RLVR turns out to be a coincidence specific to particular models, domains, and algorithms — if it fails when extended to code, science, or multimodal reasoning — then my judgment is survivorship bias from small-sample overfitting. I'll take that criticism.

The most uncomfortable inference I've already said: we may never distinguish genuine reasoning from sophisticated pattern matching. And scarier still, we may be training a cohort of "super test-takers" — models that don't "understand" math, but outperform any human at scoring high on exams of a specific format.

But even so, I still bet on "less is more."

Because real learning isn't about "how many problems you drilled," but about "whether you found that key moment of epiphany." One carefully chosen problem may be closer to the essence of education than a vast but mediocre database.

> Next time you hear about an RL project using 100K training samples, don't ask "that many?" — ask "how much worse with just one?" 💡

---

📚 Paper Details (Verified)

| Item | Content | |:---:|:---| | Title | Reinforcement Learning for Reasoning in Large Language Models with One Training Example | | Authors | Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen | | Institutions | University of Washington, USC, Microsoft, UC Santa Cruz, Georgia Tech | | arXiv ID | 2504.20571 | | Published | April 29, 2025 | | Venue | NeurIPS 2025 | | Code/Models/Data | https://github.com/ypwang61/One-Shot-RLVR | | Core method | 1-shot RLVR: RL training with a single (or few) verifiable-reward problems to probe effects on math reasoning | | Models validated | Qwen2.5-Math-1.5B / 7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B | | Algorithms validated | GRPO, PPO | | Training data | Single/double math problems (from the DeepScaleR subset) | | Benchmarks | MATH500 plus 5 other math reasoning benchmarks (6 total) | | Key results | 1-shot: MATH500 36.0%→73.6% (+8.6% non-format gain), 6-benchmark average 17.6%→35.7%; 2-shot: MATH500 74.8%, average 36.6%; on par with 1.2K-problem training | | Key findings | Post-Saturation Generalization; Cross-Domain Generalization; pure entropy loss yields +27.4%; effect stems from policy gradient loss rather than grokking |