A new paper from the University of Washington — "Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients" by Mingwei Xu and Hao Fang — challenges a core assumption of GRPO-based RLVR training: that models need to learn from both correct and incorrect answers. Their answer: negative rollouts aren't helping — and removing them entirely can *improve* performance.
The Problem with Negative Samples
GRPO (Group Relative Policy Optimization), the standard RL method behind models like DeepSeek-R1, generates groups of rollouts per problem and reinforces correct ones while penalizing incorrect ones. But the authors point out that under sparse binary rewards (1 for correct, 0 for wrong), error answers carry almost no usable signal:
> "Negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards."
Wrong answers may stem from calculation slips, misread conditions, wrong theorems, or guessing — with no consistent error structure to learn from. Penalizing a handful of sampled negatives is like inferring lottery-winning rules from a few randomly wrong tickets.
How POPO Works
POPO (Positive-Only Policy Optimization) updates the policy using only correct rollouts, with a weighted objective plus regularization terms:
Key mechanisms:
- Bounded importance weights: correct answers that are hard for the model to produce get higher weight, creating self-competition that pushes the model toward less likely correct solutions.
- Implicit negative gradients: because softmax probabilities sum to 1 (\(\sum_y \pi_\theta(y|x) = 1\)), boosting correct answers automatically suppresses incorrect ones — no explicit penalty needed.
- Siamese network + momentum anchoring: an EMA-updated anchor network prevents self-distillation collapse.
- Representation-space alignment: instead of high-variance token-level KL divergence, POPO constrains cosine similarity in the hidden embedding space.
Results
Across 5 benchmarks (MATH-500, AMC 23, AIME 24/25, OlympiadBench) and 4 base models, POPO matches or beats GRPO:
| Algorithm | MATH-500 | AMC 23 | AIME 24 | AIME 25 | Olympiad | Average | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | SFT | 84.00 | 70.55 | 32.81 | 18.93 | 44.07 | 50.07 | | GRPO | 90.80 | 85.50 | 43.33 | 30.00 | 56.38 | 61.20 | | POPO | 90.80 | 85.75 | 45.13 | 36.67 | 57.27 | 63.12 |
On Qwen-Math-7B, POPO lifts AIME 2025 from 30.00% to 36.67% (+22.2% relative). It also wins on R1-Distill-1.5B (59.92 vs 57.03 avg) and R1-Distill-7B (66.22 vs 65.60).
Ablations: adding negatives back hurts
| Configuration | AIME 25 | Olympiad | Average | |:---|:---:|:---:|:---:| | POPO (no negatives + weight reallocation) | 23.33 | 51.19 | 37.26 | | Negatives, no weight reallocation | 13.33 | 48.07 | 30.70 | | No negatives, no weight reallocation | 16.67 | 49.11 | 32.89 | | Negatives + weight reallocation | 6.67 | 39.32 | 23.00 |
Combining explicit negative penalties with POPO's implicit ones causes signal conflict — like pressing the gas and brake simultaneously — collapsing performance to 23.00. Removing either stability component (momentum adaptation or representation alignment) also degrades results (37.26 → 30.85 or 31.74).
Limitations
The authors note POPO is only validated for: 1. Sparse binary rewards (verifiable math-style right/wrong signals) — dense-reward settings are untested. 2. Text math tasks — code generation, agentic RL, and multimodal reasoning are untested. 3. Models up to 7B — behavior at 32B/70B remains open.
Paper Details
| Item | Content | |:---|:---| | Title | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Institution | University of Washington, Seattle | | arXiv | 2605.06650 | | Training data | DeepScaleR-Preview-Dataset (~40K math problems), open Qwen-based base models |
Bottom line: at least for 7B-scale math reasoning, the GPU hours spent generating negative rollouts under GRPO may be largely wasted — simply reinforcing correct answers, with the right stability mechanisms, trains better models for less compute.