A Counterintuitive Starting Point
The default consensus in reinforcement learning is that to make an AI smarter, you let it try and fail repeatedly: punish mistakes, reward successes, and the model learns what to avoid and what to pursue.
This logic sounds self-evident—until you think about wrong answers on a math exam.
A student solves a geometry proof across three pages of scratch work, then gets the area formula wrong on the final step. On another problem, he misunderstands the question from the start and spends the whole time solving a problem that doesn't exist. Both exams get zero points—but is the "error signal" the same? More importantly, can a teacher actually read "where this student's thinking went wrong" from every zero-score exam?
In LLM math-reasoning training, it's worse. GRPO (Group Relative Policy Optimization) samples a group of responses, computes average reward, and treats above-average responses as positive samples and below-average ones as negative samples. The problem: among those negatives, some only fumble the last step, some go entirely off-topic, and some are just the model "spacing out" into irrelevant content. They all get thrown into the same "bad" bucket and punished indiscriminately.
The paper's authors ask a bold question: if these negative samples can't tell a useful story, can we simply stop dealing with them?
That's the starting point of POPO (Positive-Only Policy Optimization).
---
The "Dark Forest" of Error Space
To understand why POPO abandons negative samples, consider three observations from the authors:
First, errors aren't graded. In RLVR (reinforcement learning with verifiable rewards), rewards are binary and sparse—1 for correct, 0 for wrong. On an AIME problem, a model that writes the wrong answer and a model with no idea at all receive identical feedback. Unlike a human teacher who circles "this auxiliary line was good, only the final calculation slipped," the RL reward just says "wrong." Negative samples are punished equally, but their error types differ wildly—the punishment signal contains no structural information.
Second, the error space explodes combinatorially. A math problem has a few correct paths but infinitely many wrong ones: misapplied formulas, misread conditions, skipped steps, off-topic hallucinations. Randomly sampling a few dozen negatives and hoping punishing them "covers" all error modes is like an explorer trying to understand the Amazon rainforest by stepping on three leaves.
Third, negative reasoning chains are extremely low quality. Especially during cold-start, the model fails most of the time, and failed responses often contain no coherent reasoning—just junk tokens. Learning "what not to do" from such data is like a chef learning to cook from burnt scraps: maybe you learn "this dish is inedible," but never "how to cook better."
Together these lead to a conclusion: negative samples may be noise, not signal.
---
Learn Only What's Right, Not What's Wrong
POPO's core idea is strikingly simple: during training, look only at correct responses and reinforce only successful paths.
The obvious difficulty: with no explicit punishment, won't the policy drift toward errors without a brake signal?
The authors' answer is clever: softmax applies the brake for you.
Implicit negative gradients: the softmax "probability tax"
In a POPO training step, the model generates, say, 8 responses to a problem, 3 of them correct. POPO importance-samples those 3 positives—weighting higher-probability correct answers more—and reinforces them.
The key: when gradient ascent raises the probability of those 3 correct responses, softmax normalization forces all response probabilities to sum to 1. The 5 incorrect responses must go down. No explicit punishment needed—boosting the positive automatically compresses the negative.
The authors call this the "probability tax": every time you pay bonuses to some responses, all others are taxed automatically.
But that's not enough—the compression may be too weak early in exploration. So POPO adds entropy regularization. The entropy reward keeps the policy diverse, with a side effect: when an incorrect response accidentally gains high probability, the entropy penalty specifically targets it. The most dangerous errors (highest-probability negatives) receive the strongest implicit punishment.
Theorem 3.1 shows: for any incorrect response y', the gradient of the POPO loss with respect to its logit is
∂L_POPO / ∂z_y' = π(y'|x) · [1 + β·(log π(y'|x) + H(π))]
As long as the incorrect response's probability isn't near zero, this gradient is positive—meaning the loss wants that probability to keep falling. The two terms are the probability tax (from softmax normalization of the positive-sample NLL loss) and the entropy regularization's extra penalty on high-probability errors.
POPO's theoretical core: without negative samples, negative gradients grow by themselves.
---
Three Stabilization Mechanisms
Positive-only learning is elegant but stability is a real concern—without the "anchor" of negatives, the policy can drift too far. POPO adds three mechanisms:
1. Self-competition: weight redistribution
Each positive sample gets an importance weight:
w(y|x) = π(y|x) / Z_+(x)
where Z_+(x) is the sum of probabilities over all positive samples. This creates subtle competition—if one correct response has very high probability, it "eats" a larger share within the positive set. The model must learn to distribute probability across multiple correct answers instead of putting all eggs in one basket.
Ablations confirm this: removing weight redistribution drops POPO's AIME25 score from 37.26 to 23.00. Merely "reinforcing all positives" isn't enough—internal competition among positives is key.
2. Twin network: a slightly slower self
POPO maintains two policy networks: an online policy π_θ (updated normally) and a twin policy π_ξ (slowly following via EMA momentum):
ξ ← τ·ξ + (1-τ)·θ
Imagine learning to ride a bike with a "half-second-lagging shadow" always beside you. When you attempt a radical move, the shadow pulls you back—because it's still where you were a moment ago. The twin network provides a stable, lagging but related anchor against sudden policy shifts.
Lemma 3.2 proves that under EMA momentum, the parameter gap stays bounded:
‖θ_t - ξ_t‖ ≤ τ·η·G_max / (1-τ)
So the policy never drifts too far—even without a hard KL constraint.
3. Representation-space alignment: a semantic handshake
Traditional RL uses KL divergence to keep the policy close to a reference model, but KL has problems: token-level, high variance, over-sensitive to short tokens. POPO replaces it with cosine-similarity alignment in representation space:
L_sim = -cos(h_φ(f_θ(x,y)), stop-gradient(f_ξ(x,y) + ε))
Simply put: not "output token distributions must be close" but "semantic-level vector representations must align." A predictor head h_φ (an MLP), stop-gradient, and a bit of Gaussian noise prevent shortcuts. This lets the policy freely explore at the lexical level as long as it doesn't wander in semantic space.
---
The Data: It Actually Works on Competition Math
Theory aside—results. The authors test on five math benchmarks, from easy (MATH-500) to competition level (AIME 2024/2025, Olympiad).
Standout numbers:
- Qwen-Math-7B + POPO → AIME 2025: 36.67% (GRPO: 30.00%)
- Qwen-Math-1.5B + POPO → five-benchmark average: 53.06 (GRPO: 50.22; DAPO: 50.36; Dr.GRPO: 51.22)
- R1-Distill-1.5B + POPO → five-benchmark average: 59.92
One counterexample: on R1-Distill-7B, POPO slightly underperforms GRPO. The authors' analysis: the distilled larger model's entropy drops too fast, shrinking exploration, and the distilled model's negative responses may be higher quality (R1 taught it to reason, so even its errors contain partly correct steps). This reveals POPO's sweet spot: the more "garbage" the errors, the stronger POPO; when negatives occasionally have value, GRPO is steadier.
---
Limitations and Extensions
POPO isn't a panacea. The authors candidly note:
1. Validated only on sparse binary rewards. In code generation with intermediate test feedback (dense rewards), negative samples may genuinely help—a compile error can pinpoint which line broke. 2. Tested only on math reasoning. Multimodal, code, and general dialogue remain unknowns. 3. Max 7B models. Whether larger models benefit—and why the advantage shrinks on distilled models—needs more research. 4. Positive samples can also have quality issues. If the model produces only one type of correct answer (always the same solution strategy), POPO's self-competition weights may reinforce that bias, lacking diversity.
---
Core Thesis Recap
POPO challenges a foundational RL assumption: learning requires contrast, and contrast requires presenting positives and negatives together. The authors show that under sparse binary rewards, negative samples carry too little signal, span too vast a space, and grade too coarsely—rather than punish a few random errors, reinforce successes only and let softmax normalization and entropy regularization deliver the implicit negative constraint.
The insight has broader philosophical weight: when teaching a system "what is right," the definition of "what is wrong" is sometimes embedded in "the scarcity of right." Reinforcing positive samples is itself a compression of the error space.
---
Paper Information
| Item | Detail | |------|--------| | Paper title | Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients | | Authors | Mingwei Xu, Hao Fang | | Institution | University of Washington, Seattle | | arXiv ID | arXiv:2605.06650v1 [cs.CL] | | Submitted | May 7, 2026 | | Core method | POPO: positive-only policy optimization for RLVR | | Key techniques | Bounded importance sampling, twin policy network (EMA), representation-space alignment, implicit negative gradients | | Models tested | Qwen2.5-Math series, DeepSeek-R1-Distill series, Llama-3.1-8B, DeepSeek-Math-7B | | Benchmarks | MATH-500, AMC 23, AIME 2024, AIME 2025, OlympiadBench | | Best result | Qwen-Math-7B on AIME 2025: 36.67% (GRPO 30.00%) | | Code | Not public (as of 2026-05-09) |
---
> Writing this, I wondered what Feynman himself would say about this problem. He might ask: "If you want to teach a student physics, do you show him a thousand wrong formulas, or ten correct derivations, and let him discover where the errors are?" POPO's answer: look at the correct ones first, and let errors fall out through the cracks in probability.