POPO Explained: Positive-Only Policy Optimization — When AI Learns Only From Correct Answers

Forum topic · 小凯 · 2026-05-10

Summary

This post offers a Feynman-style walkthrough of Positive-Only Policy Optimization (POPO), a reinforcement learning method proposed by Hao Fang et al. (arXiv:2605.06650) that trains LLMs using only positive (correct-answer) samples with verifiable binary rewards. The author explains why GRPO struggles in sparse-reward settings — such as AIME-level math problems where a 7B model's accuracy can be near 5%, leaving a sampled group with zero correct answers and flat, uninformative negative signals — and how POPO sidesteps this via three components: bounded importance sampling, a Siamese online/target policy network with momentum adaptation, and a bounded similarity penalty replacing KL divergence. Experiments on Qwen-Math-7B show POPO reaching 36.67% on AIME 2025 versus GRPO's 30.00%, with gains persisting on GSM8K, MATH, and AIME 2024. Ablations confirm all three components are necessary, and hyperparameter scans show robustness. The post also discusses the implicit negative gradient (raising the probability mass of correct answers necessarily reduces incorrect ones), the combinatorial vastness of answer spaces that limits negative-sample generalization, and POPO's limitations: it assumes verifiable rewards and loses value when accuracy is already high. Tags: reinforcement learning, LLM reasoning, GRPO, mathematical reasoning.

POPO Explained: When AI Learns Only From Correct Answers

*A Feynman-style walkthrough of Positive-Only Policy Optimization (POPO)*

> "If you only tell a student what's right — never what's wrong — can they still learn?" > > A paper by Hao Fang and colleagues answers yes for teaching AI to solve math problems: not only can it learn from correct answers alone, it can learn better.

Paper overview

| Field | Content | |------|---------| | Title | Positive-Only Policy Optimization (POPO) | | Authors | Hao Fang, Dong Li, Lu Tian, Jingwei Zhang, Hongyin Zhao, Ruoyi Du, Jia Li, Xiang Liu, Shuicheng Yan, Yansong Tang | | arXiv | 2605.06650 | | Date | 2026-05-08 | | Key result | Qwen-Math-7B on AIME 2025: 36.67% (GRPO: 30.00%) |

The problem: GRPO's dilemma in sparse-reward settings

Since PPO (2017), RL training has relied on comparing good and bad attempts. GRPO (DeepSeek, 2024) simplified this: for each question, generate a group of G answers, normalize rewards within the group, and reward above-average answers while penalizing below-average ones:

\hat{A}_{i} = \frac{r_i - \text{mean}({r_1, r_2, ..., r_G})}{\text{std}({r_1, r_2, ..., r_G})}

This drops the value network and is memory-efficient — but it assumes the group contains both correct and incorrect answers with informative negative signals. In math reasoning with binary rewards, a 7B model on AIME-level problems may have only ~5% accuracy, so a group of 8 samples often contains zero correct answers. Worse, all wrong answers receive the identical flat reward of 0, with no gradient of "how wrong." The author also cites combinatorial vastness: the answer space grows exponentially with reasoning steps, so penalizing a handful of sampled errors provides essentially no generalizable signal.

POPO's core idea: learn only from correct answers

POPO discards negative samples entirely. Its answer to "how do you avoid bad behavior without seeing it?" is the implicit negative gradient: probability mass sums to 1, so increasing the probability of correct answers necessarily decreases the probability of incorrect ones. Like inflating gold balloons in a crowded room — the gray ones get squeezed even if untouched.

Three technical pillars

1. Bounded importance sampling

POPO keeps only reward-1 rollouts and weights their contribution via importance sampling, with clipping to keep training stable:

w_t^{(i)} = \min\left(c, \frac{\pi_\theta(o_t^{(i)} | q, o_{<t}^{(i)})}{\pi_{\theta_{\text{old}}}(o_t^{(i)} | q, o_{<t}^{(i)})}\right)

2. Siamese policy network with momentum adaptation

A target network provides a stable reference for computing sampling ratios, updated gradually:

\theta_{\text{target}} \leftarrow \lambda \theta_{\text{target}} + (1 - \lambda) \theta_{\text{online}}

with λ close to 1 (e.g., 0.99), giving stability and a form of "memory."

3. Bounded similarity penalty replacing KL divergence

Instead of penalizing policy divergence in probability space, POPO penalizes representation distance between online and target networks:

\mathcal{L}_{\text{sim}} = \max(0, d_{\text{cos}} - \tau)

This encourages semantically similar outputs without punishing alternative valid phrasings of the same proof.

Experimental results

| Method | Qwen-Math-7B, AIME 2025 | |------|------| | GRPO | 30.00% | | POPO | 36.67% |

GSM8K: comparable to or slightly better than GRPO
MATH and AIME 2024: POPO leads

Ablations: removing bounding destabilizes training (~15% drop); removing momentum adaptation causes oscillation and GRPO-level performance; replacing the similarity penalty with KL costs 3–5% on some tasks. Robustness: stable for c ∈ [1.5, 3.0], λ ∈ [0.95, 0.999], τ ∈ [0.1, 0.3].

Why it works

Implicit negative gradient: maximizing expected reward on positives is equivalent (under probability normalization) to minimizing it on negatives.
Negative gradients fail at low accuracy: with ~5% hit rate, most GRPO groups contain no positives; GRPO ends up fitting noise from flat zero signals.
Positive answers share structure: correct reasoning paths often follow common deep patterns (e.g., construct an auxiliary line → use similar triangles → conclude), so focusing updates on positives generalizes, while penalizing scattered negatives does not — in a space with ~10^50 possible wrong paths, a thousand penalties are a drop in the ocean.

Practical guidance

| Condition | Recommended strategy | |------|---------| | Rich rewards (>50% accuracy) | Positive-negative contrast (GRPO) works | | Sparse, binary rewards (<10%) | Positive-only optimization (POPO) may win | | Sparse but continuous rewards | Keep negative samples — they carry information | | Structured error modes | Negatives valuable; design structured penalties |

Limitations

1. Verifiable rewards only: not applicable to open-ended generation where "correctness" is ambiguous. 2. Relies on sparsity: at high accuracy, negatives regain value. 3. Computational cost: importance sampling over long sequences can be expensive despite dropping the value network.

Future directions include extending POPO to continuous rewards, adaptive positive/negative mixing, and applying Siamese momentum architectures to other RL paradigms.

Takeaway

POPO echoes Skinner's finding that positive reinforcement can outperform punishment — at minimum, it shows that under verifiable, sparse-reward conditions, positive signals alone can drive efficient learning. The essence of learning, the author suggests, is not punishing errors but identifying and amplifying correctness — and when errors carry no signal, boldly ignoring them is itself an optimization strategy.

References

1. Fang, H., et al. (2026). *Positive-Only Policy Optimization*. arXiv:2605.06650. link 2. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. 4. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. 5. Skinner, B. F. (1938). *The Behavior of Organisms*. Appleton-Century-Crofts.