Your GPUs are burning. It's 2 AM, the training cluster fans are humming, and you're training a math reasoning model with GRPO—your batches full of positive and negative rollouts. That's the standard playbook, from PPO to GRPO: learn both "how to solve it" and "how to fail."
But a new paper from the University of Washington by Mingwei Xu and Hao Fang asks a deceptively simple question: are those wrong answers actually helping? Their answer is no. More strikingly, training on *correct answers only* lifted AIME 2025 scores from 30.00% to 36.67%.
> Note: GRPO (Group Relative Policy Optimization) — the mainstream RL method for training reasoning models (DeepSeek-R1, OpenAI o3). For each problem, the model generates a group of rollouts; correct and wrong answers are compared to compute a relative advantage for updates. Its hidden cost: you must generate large numbers of wrong answers to supply contrastive signal.
1. The Problem with Negative Samples: An Error Log Full of Noise
When a model produces 8 answers to a competition problem—3 right, 5 wrong—GRPO reinforces the 3 and penalizes the 5. Sounds sensible, like a student's error notebook. But the paper points out a neglected fact: wrong answers in math competitions have almost no consistent "error structure" worth teaching. A mistake may come from an arithmetic slip, a misread condition, a wrong theorem, or blind guessing—these failures share no consistent pattern.
> "Negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards."
It's like trying to infer lottery-winning patterns from five randomly filled losing tickets.
> Note: Sparse Binary Reward — reward is 1 for correct, 0 for wrong, with no intermediate credit. A zero tells the model *that* it was wrong, not *where* or *how badly*. Under such rewards, negative samples provide extremely limited signal.
2. POPO's Rebellion: Only Rewards, No Punishment
POPO (Positive-Only Policy Optimization) simply drops negative samples. Updates use only correct rollouts:
> Note: Importance Weight — POPO doesn't treat all correct answers equally. Each positive rollout gets a weight \(w_ theta(y|x)\) based on how likely the current policy is to produce it. Easy-to-find correct answers get lower weight; hard-won correct solutions get higher weight—creating a self-competition mechanism.
The core insight lies in softmax normalization:
When POPO boosts correct answers' probabilities, wrong answers are automatically squeezed—probabilities must sum to one. This is what the paper calls Implicit Negative Gradients: you don't explicitly punish wrong answers; they lose relatively when correct ones win.
> Note: Softmax Normalization and Implicit Negative Gradients — next-token probabilities come from $ pi(y|x) = frac{e^{z_y}}{ sum_{y'} e^{z_{y'}}}$. Raising the logit of a correct answer increases the denominator, lowering every other answer's relative probability. This trade-off is a mathematical necessity, no negative samples required.
3. The Data Doesn't Lie
Tested on 5 benchmarks (MATH-500, AMC 23, AIME 24, AIME 25, OlympiadBench) across 4 base models:
Qwen-Math-7B:
| Algorithm | MATH-500 | AMC 23 | AIME 24 | AIME 25 | Olympiad | Avg | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | SFT | 84.00 | 70.55 | 32.81 | 18.93 | 44.07 | 50.07 | | GRPO | 90.80 | 85.50 | 43.33 | 30.00 | 56.38 | 61.20 | | POPO | 90.80 | 85.75 | 45.13 | 36.67 🔥 | 57.27 | 63.12 |
On AIME 2025—one of the hardest benchmarks—POPO improves on GRPO's 30.00% by 22.2% relative. It also wins on R1-Distill-1.5B (59.92% vs 57.03%) and R1-Distill-7B (66.22% vs 65.60%).
All positives, no negatives, higher performance.
4. "But Without Negatives, How Does the Model Know Its Limits?"
The ablations give an uncomfortable answer:
| Configuration | AIME 25 | Olympiad | Avg | |:---|:---:|:---:|:---:| | POPO (default: no negatives + weight redistribution) | 23.33 | 51.19 | 37.26 | | Negatives + no weight redistribution | 13.33 | 48.07 | 30.70 | | No negatives + no weight redistribution | 16.67 | 49.11 | 32.89 | | Negatives + weight redistribution | 6.67 | 39.32 | 23.00 |
The most ironic row is the last: adding negative samples *back* on top of weight redistribution crashes performance to 23.00. GRPO's grouped relative advantage explicitly penalizes negatives, but POPO's weight redistribution already does the same job implicitly via softmax—stacking both creates signal conflict. Like pressing the accelerator and the brake at once.
5. A Bold Prediction
I bet that within 18 months, POPO or similar positives-only methods will become the new default for RLVR training, and GRPO's negative rollout generation will be seen as a historically legacy waste of compute.
No disrespect to GRPO's inventors—when Shao et al. proposed it in 2024, it was a huge advance that removed the value network and made RLVR practical. But each generation of solutions contains assumptions the next will break. GRPO assumed "positive-negative contrast is necessary"; POPO shows that assumption doesn't hold under sparse binary rewards. If negatives prove irreplaceable at 32B/70B scale or on code/multimodal tasks, I'll take the loss—but even then, the paper has shown that at 7B math reasoning scale, most of the wrong answers we've been burning GPU hours to generate are noise with no teaching value.
One-sentence summary of POPO: the perfect exam paper already contains all the information you need—the error notebook is a placebo.
6. Why Doesn't POPO Collapse?
Training only on correct answers sounds like an overfitting recipe, but two mechanisms prevent this:
🔒 Siamese Network + Momentum Adaptation — POPO maintains two policy networks: one updated online, one slowly updated via exponential moving average (EMA). The online network learns new tricks while the anchored network keeps it from drifting too far, breaking the risk of self-distillation collapse.
📐 Representation-Space Alignment — instead of KL divergence (token-level sensitive, high variance), POPO constrains cosine similarity in the hidden representation space:
> Note: KL Divergence vs. Representation-Space Alignment — KL measures distributional difference and is standard in RL to keep the new policy close to the old, but a single token change can swing KL wildly. POPO's alternative applies similarity constraints at the Transformer's semantic embedding level—"high-level semantic alignment" is more stable and robust than "low-level probability alignment."
Ablations confirm both components matter: removing momentum adaptation drops average performance from 37.26 to 30.85; removing representation alignment drops it to 31.74.
7. Honest Limitations
The authors list three main constraints:
1. Sparse binary rewards only: POPO is designed for math reasoning (binary right/wrong); effectiveness with dense rewards (e.g., dialogue quality scoring) is unknown. 2. Text-only math tasks: code generation, agentic RL, and multimodal reasoning are untested. 3. Max 7B scale: behavior at 32B/70B remains to be validated.
POPO isn't a panacea. It's an elegant GRPO alternative under specific conditions—verifiable, binary, sparse rewards. But those conditions are exactly today's hottest reasoning-model training scenario, and there it delivers an uncomfortable conclusion: we may have been doing redundant work all along.
8. Closing Thought
A programmer spent a week optimizing a database query from 2 seconds to 0.5. His manager asked: did the query results change? No. Then why not delete the query? Nobody uses it.
GRPO's negative rollout generation is a bit like that query—busy, expensive, and far less pedagogically valuable than assumed. POPO essentially asked: "What happens if we just delete the negatives?"
Answer: the model learns better.
Sometimes the most efficient optimization isn't adding things—it's daring to subtract. And in an industry where compute is more precious than gold, those who dare to subtract are the true engineers.
---
📚 Paper Details
| Item | Content | |:---|:---| | Title | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Institution | University of Washington, Seattle, USA | | arXiv ID | 2605.06650 | | Published | 2026-05-07 | | Category | cs.CL | | Core claim | Under RLVR's sparse binary rewards, negative rollouts yield low-quality signal; bounded importance sampling plus softmax's implicit negative gradients achieve GRPO-level or better performance using positives only | | Key techniques | Bounded Importance Sampling, Siamese Policy Network (momentum anchoring), Representation-Space Alignment (replacing KL divergence) | | Experiments | 4 base models (Qwen-Math 1.5B/7B, R1-Distill 1.5B/7B) × 5 math benchmarks (MATH-500, AMC 23, AIME 24/25, OlympiadBench) | | Key data | Qwen-Math-7B AIME 2025: POPO 36.67% vs GRPO 30.00%; ablations show re-adding negatives drops performance to 23.00% | | Code | Built on public models (Qwen family), using the DeepScaleR-Preview-Dataset (~40K math problems) |