POPO Explained: Positive-Only Policy Optimization for RL Training of LLM Math Reasoning

Forum topic · 小凯 · 2026-05-10

Summary

A Chinese forum post offers a Feynman-style explainer of Positive-Only Policy Optimization (POPO), an RL method by Hao Fang and colleagues (arXiv:2605.06650) that trains language models using only correct answers, discarding negative samples. The post argues that under verifiable binary rewards with sparse success rates (e.g., ~5% on AIME-level problems), standard group-relative methods like GRPO (DeepSeek, 2024) extract little useful signal from failed rollouts, since all wrong answers receive identical zero rewards and combinatorially vast answer spaces make negative coverage meaningless. POPO builds on three components: bounded importance sampling with a clipping constant over positive rollouts, a Siamese architecture with a momentum-updated target network for stable reference, and a bounded cosine-similarity penalty that replaces KL divergence in representation space. Reported results: Qwen-Math-7B reaches 36.67% on AIME 2025 versus GRPO's 30.00%, with gains also on GSM8K, MATH, and AIME 2024. The improvement is attributed to an 'implicit negative gradient': raising probability mass on correct answers necessarily reduces it on incorrect ones. The post also covers limitations: applicability to verifiable rewards only, reduced value when accuracy is high, and importance-sampling overhead on long sequences.

POPO Explained: Teaching AI With Only Correct Answers

This is an English translation and adaptation of a Chinese forum post introducing Positive-Only Policy Optimization (POPO) — an RL method for LLM math reasoning that learns exclusively from positive (correct) samples.

> "If you only tell a student what is right, and never what is wrong, can they still learn?"

In May 2026, Hao Fang and collaborators answered yes — at least for training AI to solve math problems. Their paper, *Positive-Only Policy Optimization* (arXiv:2605.06650), proposes a near-heretical idea: since negative samples may carry no useful signal under binary rewards, just ignore them entirely. This seemingly "spoiling" strategy outperforms mainstream methods on math reasoning benchmarks.

Paper at a glance

| Field | Content | |------|---------| | Title | Positive-Only Policy Optimization (POPO) | | Authors | Hao Fang, Dong Li, Lu Tian, Jingwei Zhang, Hongyin Zhao, Ruoyi Du, Jia Li, Xiang Liu, Shuicheng Yan, Yansong Tang | | arXiv | 2605.06650 | | Date | 2026-05-08 | | Area | Reinforcement learning · LLM reasoning · verifiable rewards | | Key result | Qwen-Math-7B on AIME 2025: 36.67% (GRPO: 30.00%) |

1. The "textbook dilemma" of RL

A 7B model on AIME-level problems may only be ~5% correct. Sampling a group of 8 answers (as GRPO does), you often get zero correct answers. GRPO (DeepSeek, 2024) simplifies PPO (OpenAI, 2017) by comparing answers within a group instead of using a value network:

\hat{A}_{i} = \frac{r_i - \text{mean}({r_1, ..., r_G})}{\text{std}({r_1, ..., r_G})}

With binary rewards (0/1), all wrong answers receive a flat, uninformative zero signal — there is no gradient of "how wrong." The paper also cites combinatorial vastness: with astronomically many possible reasoning paths, punishing a few sampled errors covers a negligible fraction of the failure space. "Ink is not enough to trace the current."

2. POPO's core idea

If negative samples don't help, learn only from positives. Via an archery analogy: instead of comparing 100 arrows (5 hits, 95 misses), POPO studies only the 5 hits and says "do more of this."

The key is the implicit negative gradient: probabilities sum to 1, so raising the probability of correct answers necessarily lowers the probability of incorrect ones. Better positive optimization automatically suppresses negative behaviors — like inflating gold balloons in a fixed-volume room, squeezing out the gray ones without touching them.

3. Three technical pillars

3.1 Bounded importance sampling

Importance sampling weights adjust positive rollouts for policy updates, with a clipping constant to prevent instability:

w_t^{(i)} = \min\left(c, \frac{\pi_\theta(o_t^{(i)} | q, o_{<t}^{(i)})}{\pi_{\theta_{\text{old}}}(o_t^{(i)} | q, o_{<t}^{(i)})}\right)

3.2 Siamese networks with momentum adaptation

An online network generates answers and gradients; a target network provides a stable reference for importance ratios, updated gradually:

\theta_{\text{target}} \leftarrow \lambda \theta_{\text{target}} + (1 - \lambda) \theta_{\text{online}}

This yields stability (no wild ratio swings) and memory of past policies.

3.3 Bounded similarity penalty instead of KL divergence

Instead of KL on probability distributions, POPO penalizes cosine distance in the Siamese representation space:

\mathcal{L}_{\text{sim}} = \max(0, d_{\text{cos}} - \tau)

This encourages semantically similar outputs without penalizing alternative valid phrasings of the same proof.

4. Experimental results

| Method | Qwen-Math-7B, AIME 2025 | |------|------| | GRPO | 30.00% | | POPO | 36.67% |

POPO also matches or beats GRPO on GSM8K, MATH, and AIME 2024. Ablations show all three components matter: removing bounded importance sampling destabilizes training (~15% drop); removing momentum adaptation causes oscillation; replacing the similarity penalty with KL drops 3–5% on some tasks. Performance is robust to hyperparameters (c ∈ 1.5–3.0, λ ∈ 0.95–0.999, τ ∈ 0.1–0.3).

5. Why it works

Signal sparsity in GRPO: at ~5% accuracy, most groups contain no positives, and zero-reward negatives fit noise.
Shared structure in positives: correct answers tend to share deep structure (e.g., construct auxiliary line → similar triangles → conclusion), enabling generalization from few samples; negative errors are combinatorially untraceable.

6. Reflections and limitations

The post draws an analogy to B.F. Skinner's finding that positive reinforcement trains behavior faster than negative, and proposes a strategy framework:

| Condition | Recommended strategy | |---------|---------------------| | Rich rewards (>50% correct) | Positive-negative contrast (GRPO) works | | Sparse binary rewards (<10%) | Positive-only optimization (POPO) may be superior | | Sparse but continuous rewards | Keep negative samples | | Structured error modes | Use structured penalties on negatives |

Limitations acknowledged in the paper: 1. Requires verifiable binary rewards — unclear for open-ended generation. 2. When accuracy is high, negatives regain value for refining details. 3. Importance sampling adds computation on very long sequences.

Future work includes extending to continuous rewards and adaptively mixing positive/negative samples based on current accuracy.

References

1. Fang, H., Li, D., Tian, L., et al. (2026). *Positive-Only Policy Optimization*. arXiv:2605.06650. 链接 2. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. 4. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. 5. Skinner, B. F. (1938). *The Behavior of Organisms*. Appleton-Century-Crofts.

---

The essence of learning, POPO suggests, is not punishing errors but identifying and amplifying what is correct. When wrong answers provide no useful information, boldly ignoring them is itself a form of policy optimization — because if you only study your missed arrows, you may never learn to hit the bullseye.