POPO: Positive-Only Policy Optimization Challenges Negative Samples in RLVR Training

Forum topic · 小凯 · 2026-05-10

Summary

POPO (Positive-Only Policy Optimization), proposed by researchers at the University of Washington, questions a foundational assumption of RLVR (Reinforcement Learning with Verifiable Rewards): that negative rollouts are necessary training signal. In sparse binary-reward settings like mathematical reasoning, negative samples suffer from missing failure-severity information and a combinatorially exploding error space. POPO trains exclusively on positive rollouts using bounded importance sampling with self-normalized weights, a Siamese policy network with EMA anchoring, and cosine-similarity representation alignment replacing KL divergence. Because softmax normalization forces all non-positive sequences' probabilities down when positive logits rise, the method produces implicit negative gradients provably equivalent to explicit penalty. Across four base models (Qwen-Math and R1-Distill, 1.5B/7B) and five math benchmarks, POPO consistently beats GRPO, with the largest gains on harder tasks (AIME 2025: 36.67% vs 30.00% for Qwen-Math-7B). Ablations show that naively combining negative samples with POPO's reweighting collapses performance from 37.26 to 23.00, indicating signal conflict between explicit penalties and implicit negative gradients. The approach may cut total training FLOPs by an estimated 50-60% by eliminating negative-rollout generation.

Background: Questioning the Negative-Sample Assumption in RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has become the mainstream paradigm for improving LLM reasoning. From PPO to GRPO, the core mechanism is contrastive learning — updating the policy via relative advantages of positive and negative rollouts. But this rests on an under-examined assumption: that negative samples provide indispensable training signal.

In sparse binary-reward settings (correct answer R=1, wrong answer R=0), negative samples face two structural problems:

| Challenge | Manifestation | Impact | |:---|:---|:---| | Missing failure severity | All wrong answers get R=0 — calculation errors, misread conditions, and bad strategies are indistinguishable | Negative samples cannot separate "almost right" from "totally off track" | | Combinatorial explosion | Wrong-answer space grows combinatorially for math problems | The few sampled negatives cannot cover the meaningful error distribution |

POPO (Positive-Only Policy Optimization), by Xu & Fang (2026), is a systematic response to this dilemma.

How POPO Works: Three Interlocking Components

1. Bounded Importance Sampling

POPO discards the separate negative rollout set entirely, defining a weighted objective over positive rollouts only. Self-normalized importance weights create a self-competition mechanism: the model is incentivized to discover solutions that are correct but unlikely under the current policy, rather than repeatedly reinforcing already-mastered paths.

2. Implicit Negative Gradients

The key mathematical insight: softmax normalization satisfies sum_y pi_theta(y|x) = 1. When gradient ascent increases a positive sample's logit, the denominator grows too, forcing the relative probability of all non-positive sequences down. The paper rigorously proves this implicit negative gradient is, in expectation, equivalent to explicitly penalizing negative samples.

3. Siamese Network and Representation Regularization

To prevent collapse from positive-only training, POPO adds:

Siamese policy network: an online network (gradient-updated) plus an anchor network updated via EMA (xi <- tau*xi + (1-tau)*theta), providing a stable evolution baseline.
Representation-space similarity constraint: instead of token-level KL divergence, a cosine-similarity penalty on Transformer hidden representations with an asymmetric prediction head, stop-gradient, and Gaussian noise — replacing common KL-style regularization.

Ablations on Qwen-Math-7B (average over benchmarks): removing weight redistribution drops 37.26 -> 32.89 (-11.7%); removing momentum anchoring 37.26 -> 30.85 (-17.2%); removing representation alignment 37.26 -> 31.74 (-14.8%).

Experimental Results

POPO was compared against GRPO on 4 base models (1.5B and 7B, math-specialized and distilled-reasoning) across MATH-500, AMC 23, AIME 24/25, and OlympiadBench:

| Model | Algorithm | MATH-500 | AMC 23 | AIME 24 | AIME 25 | Olympiad | Avg | |:---|:---|:---:|:---:|:---:|:---:|:---:|:---:| | Qwen-Math-1.5B | GRPO | 86.20 | 75.00 | 23.33 | 16.25 | 50.30 | 50.22 | | | POPO | 86.60 | 77.50 | 26.67 | 23.33 | 51.19 | 53.06 | | R1-Distill-1.5B | GRPO | 90.20 | 80.00 | 30.00 | 26.32 | 58.61 | 57.03 | | | POPO | 90.80 | 85.24 | 36.67 | 27.86 | 59.05 | 59.92 | | Qwen-Math-7B | GRPO | 90.80 | 85.50 | 43.33 | 30.00 | 56.38 | 61.20 | | | POPO | 90.80 | 85.75 | 45.13 | 36.67 | 57.27 | 63.12 | | R1-Distill-7B | GRPO | 93.60 | 87.50 | 46.67 | 33.33 | 66.91 | 65.60 | | | POPO | 93.20 | 87.25 | 47.22 | 36.67 | 66.76 | 66.22 |

Key observations:

Difficulty transfer effect: POPO's advantage grows with task difficulty — near zero on MATH-500 but +3.3% to +6.7% on AIME 2025.
Scale consistency: gains hold at both 1.5B and 7B.
Baseline independence: improvements hold regardless of whether the base model is math-specialized or a distilled reasoner.

The Most Counter-Intuitive Finding: Negative Samples Hurt

Ablations show that adding explicit negative-sample penalties on top of POPO's reweighting crashes performance from 37.26 to 23.00 (-38.3%). GRPO-style explicit punishment and POPO's implicit negative gradients conflict: when both mechanisms are active, update directions interfere and optimization destabilizes.

Implications for Training Economics

In typical GRPO training, an estimated 50-70% of rollout generation budget produces negative samples. POPO eliminates that overhead entirely; a principle-based estimate suggests total training FLOPs could drop to ~40-50% of GRPO's (rough estimate based on the paper's mechanism, not exact author-reported figures; actual savings depend on pass@k rate and group size).

Caveats:

POPO only updates when the positive set is non-empty; models with very low pass@k during cold start may stall.
The paper uses the DeepScaleR-Preview-Dataset (~40K problems) uniformly; data quality remains critical in practice.

Limitations and Future Directions

| Limitation | Scope | Extension path | |:---|:---|:---| | Reward type | Sparse binary rewards only | Adaptive rollout partitioning for dense rewards | | Task domain | Text math reasoning only | Code generation, agentic RL, multimodal reasoning | | Model scale | Up to 7B | Validation at 32B/70B |

An open question: does positive-only training limit a model's boundary-exploration ability? For tasks where understanding *why* something is wrong is required to learn *how* to be right, negative samples may remain irreplaceable.

Conclusion

POPO's core contribution is not a brand-new algorithm but a challenge to a deeply rooted default assumption in RLVR: that contrastive learning requires both positive and negative samples. Under sparse binary rewards, the mathematical structure of softmax normalization itself contains sufficient negative optimization signal — explicit negatives are not only unnecessary, but in some configurations harmful. The findings matter on three levels: efficiency (lower compute cost makes RLVR more accessible), design (a shift from contrastive to positive-reinforcement paradigms), and epistemic (re-examining every "industry default setting").

Paper Details

| Item | Content | |:---|:---| | Title | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Affiliation | University of Washington, Seattle | | arXiv | 2605.06650 | | Date | 2026-05-07 | | Training data | DeepScaleR-Preview-Dataset (~40K math problems, grade-school to olympiad level) |