Positive-Only Policy Optimization (POPO): Teaching AI to Learn Only From Correct Answers

Forum topic · 小凯 · 2026-05-10

Summary

A Chinese tech forum post presents a Feynman-style explainer of the paper "Positive-Only Policy Optimization" (POPO, arXiv:2605.06650, 2026) by Hao Fang and colleagues. The paper challenges the common belief in reinforcement learning that negative samples are necessary for training. In math reasoning tasks with binary verifiable rewards, where correctness rates can be as low as ~5%, group-based methods like GRPO often produce groups with zero correct answers and must learn from flat, uninformative zero-reward errors. POPO instead learns exclusively from positive samples, relying on an implicit negative gradient: since probabilities sum to one, raising the likelihood of correct answers automatically suppresses incorrect ones. Its three technical pillars are bounded importance sampling with clipping, a Siamese policy architecture with momentum-adapted target networks, and a bounded similarity penalty replacing KL divergence. Experiments with Qwen-Math-7B show POPO reaching 36.67% on AIME 2025 versus 30.00% for GRPO, with consistent gains on GSM8K, MATH, and AIME 2024, plus ablation studies and hyperparameter robustness checks. The post also discusses limitations, including applicability only to verifiable rewards and potential recomputation costs.

Positive-Only Policy Optimization (POPO): Teaching AI to Learn Only From Correct Answers

*An English rendering of a Feynman-style forum explainer of Positive-Only Policy Optimization (POPO)*

> "If you only tell a student what is right, and never what is wrong, can they still learn?"

In May 2026, Hao Fang and collaborators answered yes — at least for teaching AI to solve math problems — and published the result in *Positive-Only Policy Optimization* (POPO).

Paper Info

| Field | Detail | |------|--------| | Title | Positive-Only Policy Optimization (POPO) | | Authors | Hao Fang, Dong Li, Lu Tian, Jingwei Zhang, Hongyin Zhao, Ruoyi Du, Jia Li, Xiang Liu, Shuicheng Yan, Yansong Tang | | arXiv | 2605.06650 | | Date | 2026-05-08 | | Domain | Reinforcement learning, LLM reasoning, verifiable-reward optimization | | Key result | Qwen-Math-7B reaches 36.67% on AIME 2025 (GRPO: 30.00%) |

The Problem: GRPO's Dilemma with Sparse Rewards

GRPO (Group Relative Policy Optimization, DeepSeek 2024) simplified PPO by dropping the value network. For each prompt it samples a group of G answers and computes advantages relative to the group mean:

\hat{A}_{i} = \frac{r_i - \text{mean}({r_1, ..., r_G})}{\text{std}({r_1, ..., r_G})}

But in math reasoning, rewards are binary (0 or 1). A 7B model may only solve ~5% of AIME-level problems, so a group of 8 samples often contains no correct answers at all. GRPO is then forced to learn from all-wrong groups, where every error yields the same flat "0" signal with no gradient of "degree of failure." Moreover, the answer space suffers from combinatorial vastness: punishing a few sampled wrong answers is like dropping ink into the ocean to trace the current — the negative signal does not generalize to astronomically many other failure modes.

POPO's Core Idea

POPO learns only from positive samples — like an archery coach who analyzes only the 5 arrows that hit the bullseye rather than the 95 misses. Raising the probability of correct answers automatically lowers the probability of wrong ones, since total probability must sum to 1. This is the paper's implicit negative gradient: positive reinforcement inherently suppresses negative behavior.

Three Technical Pillars

1. Bounded Importance Sampling

POPO keeps only reward-1 rollouts and reweights them via importance sampling, with a clipping bound $$c$$ to keep training stable:

w_t^{(i)} = \min\left(c, \frac{\pi_\theta(o_t^{(i)} | q, o_{<t}^{(i)})}{\pi_{\theta_{\text{old}}}(o_t^{(i)} | q, o_{<t}^{(i)})}\right)

A single lucky positive sample cannot dominate the update.

2. Siamese Policy Networks with Momentum Adaptation

POPO maintains an online network and a momentum-updated target network:

\theta_{\text{target}} \leftarrow \lambda \theta_{\text{target}} + (1 - \lambda) \theta_{\text{online}}

with $\lambda$ close to 1 (e.g., 0.99). The target network provides a stable reference frame for importance-sampling ratios, avoiding oscillation.

3. Bounded Similarity Penalty Instead of KL Divergence

Rather than penalizing policy divergence in probability space via KL, POPO penalizes representational divergence in the Siamese embedding space:

\mathcal{L}_{\text{sim}} = \max(0, d_{\text{cos}} - \tau)

This encourages semantically similar answers rather than identical token sequences — important when the same proof can be expressed in many valid ways.

Results

| Method | Qwen-Math-7B on AIME 2025 | |--------|---------------------------| | GRPO | 30.00% | | POPO | 36.67% |

POPO matches or beats GRPO on GSM8K, MATH, and AIME 2024 as well. Ablations show all three components are necessary:

Removing the bounding on importance sampling destabilizes training (~15% performance drop).
Removing momentum adaptation causes severe oscillation, with performance falling back to GRPO levels.
Replacing the similarity penalty with KL divergence costs 3–5% on some tasks.

Hyperparameter sweeps show robustness: $$c$$ stable in 1.5–3.0, $\lambda$ in 0.95–0.999, $\tau$ in 0.1–0.3.

Why It Works

Implicit negative gradient: maximizing expected reward over positive samples, under probability normalization, is equivalent to suppressing negative ones — like inflating golden balloons in a fixed-volume room, which inevitably squeezes the gray ones.
GRPO fails at low pass rates: with ~5% correctness, most groups have no positive anchor, and flat zero-reward errors amount to fitting noise.
Positive samples share structure: correct reasoning paths tend to follow deep common patterns, so focusing on them enables generalization, whereas negative penalties do not cover the vast failure space.

Limitations and Future Directions

The authors note POPO:

1. Requires verifiable binary rewards — it does not apply to open-ended generation like creative writing. 2. Depends on sparsity: when correctness exceeds ~50%, negative samples may regain informational value. 3. Incurs importance-sampling overhead on very long sequences despite dropping the value network.

Suggested future work includes extending POPO to continuous rewards, adaptively choosing the positive/negative ratio based on current pass rate, and applying the Siamese momentum architecture to other RL paradigms.

References

1. Fang, H., Li, D., Tian, L., et al. (2026). *Positive-Only Policy Optimization*. arXiv:2605.06650. Link 2. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. 4. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. 5. Skinner, B. F. (1938). *The Behavior of Organisms*. Appleton-Century-Crofts.

The takeaway: when wrong answers carry no usable information, ignoring them and amplifying the correct ones is not spoiling the student — it is strategy optimization.

Positive-Only Policy Optimization (POPO): Teaching AI to Learn Only From Correct Answers

Summary

Positive-Only Policy Optimization (POPO): Teaching AI to Learn Only From Correct Answers

Paper Info

The Problem: GRPO's Dilemma with Sparse Rewards

POPO's Core Idea

Three Technical Pillars

1. Bounded Importance Sampling

2. Siamese Policy Networks with Momentum Adaptation

3. Bounded Similarity Penalty Instead of KL Divergence

Results

Why It Works

Limitations and Future Directions

References

Tags