POPO: Positive-Only Policy Optimization — Teaching AI Math from 'Excellent Essays' Instead of Punishing Mistakes

Forum topic · 小凯 · 2026-05-11

Summary

This forum post explains POPO (Positive-Only Policy Optimization), a reinforcement learning method for improving LLM math reasoning, through a classroom analogy. Traditional RLVR methods like PPO and GRPO reward correct answers and penalize incorrect ones among sampled rollouts. The post argues that with sparse binary rewards, penalizing negatives is nearly meaningless: all wrong answers receive identical zero reward regardless of how close they were, and the combinatorially vast space of possible errors means punishing a few sampled negatives cannot cover the long tail of unseen failure modes. Citing authors Mingwei Xu and Hao Fang (University of Washington), the post describes POPO's alternative: ignore negative rollouts entirely and train only on correct answers, comparing them against each other so the model learns what 'better correctness' looks like. Un-enforced directions simply fall away without explicit penalties. The post positions this as an intuitive, counter-conventional insight: correctness can be defined without contrast against errors.

🎯 Only Learning from Top Students: How POPO Teaches AI Math from 'Excellent Essays'

> *"Don't try to define correctness through mistakes. The right path doesn't need wrong ones to prove its value."* — A made-up quote that captures a counter-intuitive truth.

---

🏫 Two Teachers, Two Teaching Styles

Imagine you are a math teacher facing a group of struggling students. You have two possible strategies.

Strategy A: Traditional Grading

You give students 100 problems. They get 12 right and 88 wrong. You tell them:

"Let's look at the 12 correct answers — these are positive examples to learn from. Now the 88 wrong ones — you must understand where you went wrong."

This is the standard reinforcement learning approach. PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) both work this way: the model generates a set of rollouts, some correct (positive), some wrong (negative). Reward the correct ones, penalize the wrong ones, and steer the model toward correctness through contrast.

Sounds reasonable, right? Mistakes are part of learning.

Strategy B: The 'Excellent Essays' Method

Another teacher uses a completely different approach. Given the same 100 problems, he focuses *only* on the 12 correct answers.

"Today we won't analyze mistakes. We'll only study these 12 correct solutions. Zhang San's approach is especially clever — he used auxiliary lines. Li Si's is elegant, one step to the answer. Wang Wu's is long but every step is solid. Let's compare these correct solutions — which is more elegant? Which is more general?"

This teacher never looks at wrong answers. His theory: the 'error patterns' among 88 wrong answers are nearly infinite — calculation slips, conceptual confusion, typos, total misunderstanding. Analyzing them is like fishing for needles in the ocean.

But the 12 correct answers? Those *can* be compared. Some solutions are better than others. By comparing within the set of correct solutions, students naturally infer what 'better correctness' means. Approaches absent from the 'excellent solutions' set implicitly represent inferior directions — no explicit punishment needed.

This is the core idea of POPO (Positive-Only Policy Optimization).

---

🎲 Why 'Punishing Mistakes' May Be Pointless

Before diving into POPO, we must understand a key problem: in the RLVR (reinforcement learning with verifiable rewards) setting, what exactly is wrong with negative samples?

RLVR is the dominant recent training paradigm for improving LLM reasoning. The basic logic: generate many answers, score them with a verifiable reward function (e.g., whether a math answer is correct), give +1 for correct and 0 for wrong, then update the model.

GRPO is the current star of this paradigm. For each question, the model generates a group of answers (e.g., 8), then compares them — answers above the group average are rewarded, those below are penalized. It seems simpler and more efficient than PPO (no extra value network), with surprisingly strong results.

But POPO's two authors (Mingwei Xu and Hao Fang, both from the University of Washington) point out an issue GRPO hasn't fully addressed:

Under sparse binary rewards, penalizing negative samples is nearly meaningless.

Why? Consider a math competition problem with a numeric answer. The model generates 8 answers, 2 correct (say, 42) and 6 wrong (some say 38, some 100, some are gibberish).

GRPO rewards the 2 correct ones and penalizes the 6 wrong ones. But wait — is there any difference *among* those 6 wrong answers? From the reward function's perspective, no. Whether an answer is 'off by a little' or 'completely absurd', it receives the same 0-score penalty. An answer of '41' (off by 1) and an answer of 'banana' (totally irrelevant) are equal before punishment.

As the paper's authors put it:

> "Negative rollouts may admit no gradation of failure severity."

Even more fatally, there's a second problem:

> "The combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards."

How large is the output space of math reasoning? Essentially infinite. A math problem admits infinitely many wrong solutions. You sample 6 negatives and penalize them, but infinitely many errors remain unseen. Punish 6 known mistakes, and the model may make the 7th, 100th, or 10,000th next time — you cannot cover all 'unknown errors' by 'punishing known errors'.

It's like trying to maintain public safety by 'listing all bad people' — the list never ends. POPO's approach: cultivate the good ones, making the standard of 'good' increasingly clear and high — those who don't meet it are naturally eliminated.

---

*(This analysis is published in multiple parts; see replies below for the continuation.)*

#PapersCool #DailyPaper #ReinforcementLearning #POPO