A Counterintuitive Starting Point
The default consensus in reinforcement learning is that AI must learn from trial and error: punish mistakes, reward successes, and the model learns what to avoid and what to pursue.
This logic sounds self-evident—until you think about wrong answers on a math exam. A student solves a geometry proof across three pages of scratch work but botches the area formula on the last step. On another problem, they misunderstand the question from the start and solve a problem that doesn't exist. Both scores are zero, but is the "error signal" the same? Can a teacher really extract "where the thinking went wrong" from every zero-score paper?
In LLM math reasoning training, the situation is worse. GRPO (Group Relative Policy Optimization) samples a group of responses, computes an average reward, and treats above-average answers as positive and below-average as negative. But among those negatives, some merely miscalculated the final step, some went completely off-topic, and some are just incoherent token noise. All get dumped into the same "bad" bucket and punished indiscriminately.
The authors ask a bold question: if these negative samples tell no useful story, can we simply stop dealing with them?
That is the starting point of POPO (Positive-Only Policy Optimization).
---
The "Dark Forest" of the Error Space
Three observations motivate abandoning negative samples:
1. Errors are ungraded. In RLVR (reinforcement learning with verifiable rewards), rewards are binary and sparse: 1 for correct, 0 for wrong. On an AIME problem, a nearly-correct answer and a clueless one receive identical feedback. Unlike a human teacher who circles "good auxiliary line, just a calculation slip," the reward function only says "wrong." All negatives are punished equally, but their error types differ wildly—the punishment signal carries no structural information.
2. The error space is combinatorially explosive. A math problem has a handful of correct solution paths but infinitely many roads to wrong answers: wrong formula, misread condition, skipped step, or pure hallucinated content. Randomly sampling a few dozen negatives and hoping punishment covers all error modes is like an explorer trying to understand the Amazon rainforest by stepping on three leaves.
3. Negative chains of thought are extremely low-quality. Especially during cold start, most model outputs are wrong, and failed responses often contain no coherent reasoning—just garbage tokens. Learning "what not to do" from such data is like a chef learning cooking from burnt residue: you might learn "this dish is inedible," but never "how to cook better."
Together, these imply: negative samples may be noise, not signal.
---
Learn Only from What's Right
POPO's core idea is disarmingly simple: during training, look only at correct responses; reinforce only successful paths.
The obvious worry: without negative penalties, won't the policy drift toward errors with no braking signal?
The authors' answer: softmax brakes for you. This is the paper's most elegant insight.
Implicit Negative Gradients: The "Probability Tax"
Suppose a model generates 8 responses to a problem and 3 are correct. POPO performs importance sampling on those 3 positives, weighting higher-probability correct answers more heavily and reinforcing them.
The key: when gradient ascent raises the probability of the 3 positives, softmax normalization forces all response probabilities to sum to 1. The 5 incorrect responses' probabilities must fall. No explicit punishment needed—boosting the positive automatically compresses the negative.
The authors call this a "probability tax": every time bonuses are paid to some responses, all others are automatically taxed.
Entropy regularization adds a second mechanism. It encourages diversity, but as a side effect it "attends" to incorrect answers that accidentally gain high probability—meaning the most dangerous errors (highest-probability negatives) receive the strongest implicit punishment.
Theorem 3.1 proves that for any incorrect response y', the POPO loss gradient with respect to its logit is
∂L_POPO / ∂z_y' = π(y'|x) · [1 + β·(log π(y'|x) + H(π))]
As long as the error's probability isn't near zero, this gradient is positive—the loss wants that probability to keep falling. The first term is the probability tax (from softmax normalization of the positive-sample NLL loss); the second is entropy regularization's extra penalty on high-probability errors.
This is POPO's theoretical core: without negative samples, negative gradients grow on their own.
---
Three Stabilization Mechanisms
Learning only from positives is elegant, but stability is a practical concern. Without the "anchoring" of negatives, policies can drift too far. POPO adds three mechanisms:
1. Self-Competition: Weight Redistribution
Each positive gets an importance weight:
w(y|x) = π(y|x) / Z_+(x)
where Z_+(x) is the sum of positive-sample probabilities. This creates subtle competition—if one correct answer's probability is especially high, it "eats" a larger share within the positive set. The model must learn to distribute probability across multiple correct answers rather than putting all eggs in one basket.
Ablations confirm this: removing weight redistribution drops POPO's AIME25 score from 37.26 to 23.00. Merely "reinforcing all positives" isn't enough—internal competition among positives is key.
2. Twin Network: A Slightly-Slower Self
POPO maintains two policy networks: an online policy π_θ (updated normally) and a twin policy π_ξ that follows slowly via EMA momentum:
ξ ← τ·ξ + (1-τ)·θ
Like learning to bicycle with a slightly-lagging shadow beside you that holds you back when you attempt a radical move. The twin provides a stable, lagged but correlated anchor against policy shocks.
Lemma 3.2 proves the parameter gap is always bounded:
‖θ_t - ξ_t‖ ≤ τ·η·G_max / (1-τ)
The policy never strays too far—even without a hard KL constraint.
3. Representation-Space Alignment: A Semantic Handshake
Traditional RL uses KL divergence to keep the policy close to a reference model, but KL is token-level, high-variance, and overly sensitive to short tokens. POPO replaces it with cosine-similarity alignment in representation space:
L_sim = -cos(h_φ(f_θ(x,y)), stop-gradient(f_ξ(x,y) + ε))
In short: instead of "output token distributions must be similar," it's "semantic-level vector representations must align." A predictor head h_φ (an MLP), stop-gradient, and slight Gaussian noise prevent shortcut-taking. This lets the policy explore freely at the lexical level, as long as it doesn't drift in semantic space.
---
Results: It Actually Works on Competition Math
The authors benchmark on five math reasoning datasets, from easy (MATH-500) to competition level (AIME 2024/2025, OlympiadBench).
Highlights:
- Qwen-Math-7B + POPO → AIME 2025: 36.67% (GRPO: 30.00%)
- Qwen-Math-1.5B + POPO → five-benchmark average: 53.06 (GRPO: 50.22; DAPO: 50.36; Dr.GRPO: 51.22)
- R1-Distill-1.5B + POPO → five-benchmark average: 59.92
One counterexample: on R1-Distill-7B, POPO slightly underperforms GRPO. The authors suggest distilled larger models lose entropy too quickly, shrinking the exploration space, and distilled models' negative responses may be higher-quality (R1 taught them to reason, so even errors contain partially correct steps). This clarifies POPO's适用 range: the garbage-ier the errors, the stronger POPO; when negatives occasionally carry value, GRPO is safer.
---
Limitations and Extensions
POPO is not a panacea. The authors candidly note boundaries:
1. Validated only on sparse binary rewards. In code generation with intermediate test feedback (dense rewards), negative samples may genuinely help—a compilation error pinpoints which line broke.
2. Tested only on math reasoning. Multimodal, code, and general dialogue remain open questions.
3. Maximum 7B models. Whether larger models benefit—and why the advantage shrinks on distilled models—needs more study.
4. Positives can have quality issues too. If the model produces only one kind of correct answer (always the same solution path), POPO's self-competition weights reinforce that bias, lacking diversity.
---
Core Argument Recap
POPO challenges a foundational RL assumption: learning must rely on contrast, and contrast requires both positive and negative examples. The authors show that under sparse binary rewards, negative samples are too low-signal, the error space too vast, and grading too coarse. Rather than punishing a few random errors, reinforce successes only—let softmax normalization and entropy regularization impose the implicit negative constraints.
The insight carries broader philosophical weight: when you teach a system "what is right," the definition of "what is wrong" is sometimes embedded in the scarcity of what's right. Reinforcing positives is itself a compression of the error space.
---
Paper Information
| Item | Detail | |------|--------| | Paper | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Institution | University of Washington, Seattle | | arXiv ID | arXiv:2605.06650v1 [cs.CL] | | Submitted | May 7, 2026 | | Method | POPO: positive-only RLVR policy optimization | | Key techniques | Bounded importance sampling, twin EMA policy networks, representation-space alignment, implicit negative gradients | | Models | Qwen2.5-Math series, DeepSeek-R1-Distill series, Llama-3.1-8B, DeepSeek-Math-7B | | Benchmarks | MATH-500, AMC 23, AIME 2024, AIME 2025, OlympiadBench | | Best result | Qwen-Math-7B on AIME 2025: 36.67% (GRPO 30.00%) | | Code | Not publicly released (as of 2026-05-09) |
> A closing thought in the Feynman spirit: "If you're teaching a student physics, do you show them a thousand wrong formulas, or ten correct derivations and let them discover where errors live?" POPO's answer: show the correct ones first, and let errors fall through the gaps in probability on their own.