POPO: Positive-Only Policy Optimization Beats GRPO on Math Reasoning — Negative Rollouts May Be Waste

Forum topic · 小凯 · 2026-05-10

Summary

A new paper from the University of Washington by Mingwei Xu and Hao Fang, 'Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients' (arXiv:2605.06650), challenges a core assumption of GRPO-based RLVR training: that negative rollouts are needed for learning. The authors argue that under sparse binary rewards, erroneous answers carry little consistent teaching signal. Their method, POPO (Positive-Only Policy Optimization), trains exclusively on correct rollouts, using bounded importance sampling to upweight rare-but-correct solutions, while softmax normalization implicitly suppresses incorrect answers ('implicit negative gradients'). Stability is maintained via a siamese policy network with momentum anchoring and a representation-space cosine alignment loss replacing KL divergence. Across five math benchmarks and four base models up to 7B, POPO matches or beats GRPO: on AIME 2025 with Qwen-Math-7B it scores 36.67% versus GRPO's 30.00%. Ablations show that re-adding negative samples actually degrades performance. The paper's limitations: only sparse binary rewards, text-only math tasks, and models up to 7B were tested.

Key Idea

A new paper from the University of Washington — *Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients* by Mingwei Xu and Hao Fang (arXiv:2605.06650) — asks a deceptively simple question: do negative rollouts actually help in GRPO-style RLVR training? Their answer is no. Training only on correct answers (POPO, Positive-Only Policy Optimization) matches or beats GRPO — e.g., lifting Qwen-Math-7B's AIME 2025 score from 30.00% to 36.67%.

Why negative rollouts are noise

Under sparse binary rewards (1 for correct, 0 for wrong), failed answers have no gradation of failure severity — they may stem from arithmetic slips, misread conditions, wrong theorems, or random guessing, with no consistent structure. Penalizing a few sampled negatives covers almost no meaningful reward signal.

How POPO works without negatives

Positive-only updates with bounded importance sampling: each correct rollout gets a weight $w_\theta(y|x)$ ; solutions the model rarely finds are upweighted, creating a self-competition dynamic.
Implicit negative gradients: because softmax normalizes probabilities to sum to 1, raising the probability of correct answers automatically squeezes down incorrect ones — no explicit penalty needed.
Siamese network + momentum anchoring: an EMA-updated target network prevents self-distillation collapse.
Representation-space alignment: instead of token-level KL divergence (noisy, high-variance), POPO uses a cosine-similarity constraint on hidden representations.

\mathcal{L}_{\text{POPO}} = -\mathbb{E}_{x \sim D}\left[\sum_{y \in S^+(x)} w_\theta(y|x) \cdot \log \pi_\theta(y|x)\right] + \alpha \mathcal{L}_{\text{sim}} + \beta \mathcal{L}_{\text{ent}}

Results (Qwen-Math-7B)

| Algorithm | MATH-500 | AMC 23 | AIME 24 | AIME 25 | OlympiadBench | Avg | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | SFT | 84.00 | 70.55 | 32.81 | 18.93 | 44.07 | 50.07 | | GRPO | 90.80 | 85.50 | 43.33 | 30.00 | 56.38 | 61.20 | | POPO | 90.80 | 85.75 | 45.13 | 36.67 | 57.27 | 63.12 |

POPO also wins on R1-Distill-1.5B (59.92% vs 57.03%) and R1-Distill-7B (66.22% vs 65.60%).

The surprising ablation

Adding negative samples *back* hurts: with negatives plus weight redistribution, average performance crashes from 37.26 to 23.00. POPO's implicit suppression already does GRPO's job — combining both creates conflicting signals, like pressing the accelerator and brake simultaneously.

Stated limitations

1. Designed for sparse binary rewards only; dense-reward settings are untested. 2. Text-only math tasks; code generation, agentic RL, and multimodal reasoning untested. 3. Tested only up to 7B models; 32B/70B behavior unknown.

Paper details

Title: Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
Authors: Mingwei Xu, Hao Fang (University of Washington, Seattle)
arXiv: 2605.06650, cs.CL
Training setup: public Qwen/R1-Distill models, DeepScaleR-Preview-Dataset (~40K math problems)

Bottom line: for verifiable, binary-reward math reasoning at 7B scale, most GPU hours spent generating negative rollouts may be producing noise with little teaching value. Sometimes the best optimization is subtraction.