Positive-Only Policy Optimization (POPO): Teaching LLMs to Reason From Correct Answers Alone

*An English adaptation of a Feynman-style explainer originally published on zhichai.net.*

> "If you only tell a student what is right, and never what is wrong, can they still learn?" A 2026 paper by Hao Fang et al. answers yes—at least for teaching AI to solve math problems—and claims it can even work better.

Paper overview

| Field | Detail | |------|--------| | Title | Positive-Only Policy Optimization (POPO) | | Authors | Hao Fang, Dong Li, Lu Tian, Jingwei Zhang, Hongyin Zhao, Ruoyi Du, Jia Li, Xiang Liu, Shuicheng Yan, Yansong Tang | | arXiv | 2605.06650 | | Date | 2026-05-08 | | Domain | Reinforcement learning, LLM reasoning, verifiable-reward optimization | | Key result | Qwen-Math-7B scores 36.67% on AIME 2025 (GRPO: 30.00%) |

The problem with GRPO

GRPO (Group Relative Policy Optimization), introduced by DeepSeek in 2024, simplified PPO by dropping the value network. For each prompt it samples a group of G answers and computes advantages by group-relative normalization:

where rewards \(r_i\) are typically binary (0 or 1). But in math reasoning, a 7B model may only solve ~5% of AIME-level problems. With 8 samples per group, that's 0.4 correct answers on average—often zero. GRPO is then forced to learn from incorrect answers, yet:

• No severity gradient: under binary rewards, all wrong answers receive the same flat signal of 0. "Almost right" and "totally off" are indistinguishable.
• Combinatorial vastness: the answer space grows exponentially with reasoning steps. Penalizing a handful of sampled mistakes cannot cover it—*ink is not enough to trace the current*.

POPO's idea: learn only from correct answers

POPO's philosophy: since negative samples provide no useful signal, train only on positive rollouts. Its archery metaphor: GRPO lays out all 100 arrows (5 hits, 95 misses) and says "these 5 are better than those 95"; POPO takes only the 5 bullseyes and says "do more of this."

Why does this work? Through the implicit negative gradient: probability mass is conserved, so increasing the probability of correct answers automatically decreases that of incorrect ones. Raising the hit rate from 5% to 10% inherently reduces the miss rate from 95% to 90%. The post calls this "rollout redistribution"—maximizing expected reward on positives is equivalent (under normalization) to minimizing it on negatives.

Three technical pillars

1. Bounded importance sampling

POPO keeps only reward-1 rollouts and reweights them by the new/old policy probability ratio, clipped at a constant \(c\):

\[w_t^{(i)} = \min\left(c, \frac{\pi_\theta(o_t^{(i)} | q, o_{<t}^{(i)})}{\pi_{\theta_{\text{old}}}(o_t^{(i)} | q, o_{<t}^{(i)})}\right)\]

Without clipping, an outlier positive sample could dominate the update and destabilize training.

2. Siamese policy network with momentum adaptation

POPO maintains an online network (generates answers, computes gradients) and a target network (a stable reference for importance ratios), updated gradually:

\[\theta_{\text{target}} \leftarrow \lambda \theta_{\text{target}} + (1 - \lambda) \theta_{\text{online}}\]

with \(\lambda\) near 1 (e.g., 0.99). This keeps ratios in a sane range and gives training a form of memory, compared by the authors to gradual natural selection in policy evolution.

3. Bounded similarity penalty instead of KL divergence

Rather than a KL penalty on token distributions, POPO penalizes cosine distance between the two networks' semantic representations:

\[\mathcal{L}_{\text{sim}} = \max(0, d_{\text{cos}} - \tau)\]

This encourages semantically similar outputs without punishing alternative-but-equivalent phrasings of the same proof.

Results

AIME 2025 (Qwen-Math-7B):

| Method | Accuracy | |--------|----------| | GRPO | 30.00% | | POPO | 36.67% |

POPO also matches or beats GRPO on GSM8K, MATH, and AIME 2024. Ablations: removing bounded importance sampling destabilizes training (~15% drop); removing momentum adaptation causes oscillation and GRPO-level performance; reverting to KL divergence costs 3–5% on some tasks. Performance is stable across \(c \in [1.5, 3.0]\), \(\lambda \in [0.95, 0.999]\), and \(\tau \in [0.1, 0.3]\).

Why "only positives" works

• Sparse binary rewards make negatives uninformative: with ~5% success, most GRPO groups contain all-zero rewards—no contrast signal, only noise.
• Positive samples share deep structure: correct proofs often follow common patterns (e.g., construct auxiliary lines → use similar triangles → conclude), so focusing on them enables generalization, while penalty signals over a \(10^{50}\)-sized error space do not.

Implications and limitations

The post suggests a decision framework: with rich rewards (>50% success), positive-negative contrast (GRPO) works well; with sparse binary rewards (<10%), positive-only optimization may be superior; with sparse but continuous rewards, negatives retain information. It also notes an echo of Skinner's finding that positive reinforcement trains behavior faster than negative reinforcement.

Limitations acknowledged by the authors: 1. POPO assumes verifiable binary rewards—it does not apply to open-ended generation. 2. If success rates are already high, negatives regain value for fine-grained improvement. 3. Importance sampling adds computational overhead on very long sequences.

Possible future work: extending to continuous rewards, adaptive positive/negative ratios based on current success rate, and applying Siamese momentum architectures to other RL paradigms.

References

1. Fang, H., Li, D., Tian, L., et al. (2026). *Positive-Only Policy Optimization*. arXiv:2605.06650. link 2. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. 3. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. 4. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. 5. Skinner, B. F. (1938). *The Behavior of Organisms*. Appleton-Century-Crofts.