POPO: Training LLMs with Only Positive Rollouts Beats GRPO on Math Reasoning

Forum topic · 小凯 · 2026-05-10

Summary

A University of Washington paper by Mingwei Xu and Hao Fang introduces POPO (Positive-Only Policy Optimization), a reinforcement learning method for reasoning models that discards negative rollouts entirely. Under sparse binary rewards typical of math reasoning, the authors argue negative answers carry little usable signal because failure modes lack consistent structure. POPO updates the policy using only correct answers with bounded importance weights, relying on softmax normalization to create implicit negative gradients that suppress wrong answers automatically. Stability is maintained via a Siamese policy network with momentum (EMA) anchoring and representation-space cosine-similarity alignment instead of KL divergence. Across five benchmarks (MATH-500, AMC 23, AIME 24/25, OlympiadBench) and four base models, POPO matches or exceeds GRPO: Qwen-Math-7B improves AIME 2025 accuracy from 30.00% to 36.67%, a 22.2% relative gain. Ablations show adding negative samples back degrades performance, dropping average scores from 37.26 to as low as 23.00. The paper (arXiv:2605.06650) is limited to binary rewards, text math tasks, and models up to 7B, but suggests GRPO's costly negative rollout generation may be largely wasted compute in RLVR training.

A new paper from the University of Washington — "Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients" by Mingwei Xu and Hao Fang — challenges a core assumption of GRPO-based RLVR training: that models need to learn from both correct and incorrect answers. Their answer: negative rollouts aren't helping — and removing them entirely can *improve* performance.

The Problem with Negative Samples

GRPO (Group Relative Policy Optimization), the standard RL method behind models like DeepSeek-R1, generates groups of rollouts per problem and reinforces correct ones while penalizing incorrect ones. But the authors point out that under sparse binary rewards (1 for correct, 0 for wrong), error answers carry almost no usable signal:

> "Negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards."

Wrong answers may stem from calculation slips, misread conditions, wrong theorems, or guessing — with no consistent error structure to learn from. Penalizing a handful of sampled negatives is like inferring lottery-winning rules from a few randomly wrong tickets.

How POPO Works

POPO (Positive-Only Policy Optimization) updates the policy using only correct rollouts, with a weighted objective plus regularization terms:

\mathcal{L}_{\text{POPO}} = -\mathbb{E}_{x \sim D} \left[ \sum_{y \in S^+(x)} w_\theta(y|x) \cdot \log \pi_\theta(y|x) \right] + \alpha \mathcal{L}_{\text{sim}} + \beta \mathcal{L}_{\text{ent}}

Key mechanisms:

Bounded importance weights: correct answers that are hard for the model to produce get higher weight, creating self-competition that pushes the model toward less likely correct solutions.
Implicit negative gradients: because softmax probabilities sum to 1 ( $\sum_y \pi_\theta(y|x) = 1$ ), boosting correct answers automatically suppresses incorrect ones — no explicit penalty needed.
Siamese network + momentum anchoring: an EMA-updated anchor network prevents self-distillation collapse.
Representation-space alignment: instead of high-variance token-level KL divergence, POPO constrains cosine similarity in the hidden embedding space.

Results

Across 5 benchmarks (MATH-500, AMC 23, AIME 24/25, OlympiadBench) and 4 base models, POPO matches or beats GRPO:

| Algorithm | MATH-500 | AMC 23 | AIME 24 | AIME 25 | Olympiad | Average | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | SFT | 84.00 | 70.55 | 32.81 | 18.93 | 44.07 | 50.07 | | GRPO | 90.80 | 85.50 | 43.33 | 30.00 | 56.38 | 61.20 | | POPO | 90.80 | 85.75 | 45.13 | 36.67 | 57.27 | 63.12 |

On Qwen-Math-7B, POPO lifts AIME 2025 from 30.00% to 36.67% (+22.2% relative). It also wins on R1-Distill-1.5B (59.92 vs 57.03 avg) and R1-Distill-7B (66.22 vs 65.60).

Ablations: adding negatives back hurts

| Configuration | AIME 25 | Olympiad | Average | |:---|:---:|:---:|:---:| | POPO (no negatives + weight reallocation) | 23.33 | 51.19 | 37.26 | | Negatives, no weight reallocation | 13.33 | 48.07 | 30.70 | | No negatives, no weight reallocation | 16.67 | 49.11 | 32.89 | | Negatives + weight reallocation | 6.67 | 39.32 | 23.00 |

Combining explicit negative penalties with POPO's implicit ones causes signal conflict — like pressing the gas and brake simultaneously — collapsing performance to 23.00. Removing either stability component (momentum adaptation or representation alignment) also degrades results (37.26 → 30.85 or 31.74).

Limitations

The authors note POPO is only validated for: 1. Sparse binary rewards (verifiable math-style right/wrong signals) — dense-reward settings are untested. 2. Text math tasks — code generation, agentic RL, and multimodal reasoning are untested. 3. Models up to 7B — behavior at 32B/70B remains open.

Paper Details

| Item | Content | |:---|:---| | Title | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Institution | University of Washington, Seattle | | arXiv | 2605.06650 | | Training data | DeepScaleR-Preview-Dataset (~40K math problems), open Qwen-based base models |

Bottom line: at least for 7B-scale math reasoning, the GPU hours spent generating negative rollouts under GRPO may be largely wasted — simply reinforcing correct answers, with the right stability mechanisms, trains better models for less compute.