Positive-Only Learning: How POPO Teaches AI Math from 'Excellent Essays'

Forum topic · 小凯 · 2026-05-11

Summary

This zhichai.net forum post explains POPO (Positive-Only Policy Optimization), a reinforcement learning method for LLM reasoning introduced by Mingwei Xu and Hao Fang of the University of Washington. Using a teaching analogy, the author contrasts standard RL approaches like PPO and GRPO—which reward correct rollouts and penalize incorrect ones—with POPO's strategy of learning exclusively from correct answers. The post argues that under sparse binary rewards in RLVR (Reinforcement Learning with Verifiable Rewards), penalizing negative samples is largely meaningless: all wrong answers receive identical zero reward regardless of how close they were to being correct, and the combinatorial vastness of possible failures means sampled negatives cannot cover meaningful reward signal. Instead, POPO compares only correct solutions against each other, letting the model infer what 'better' looks like without explicitly punishing failures. Full multi-part analysis continues in forum replies.

Positive-Only Learning: How POPO Teaches AI Math from "Excellent Essays"

> *"Don't try to define correctness through errors. The correct path doesn't need wrong examples to prove its value."* — The author admits this quote is invented, but says it captures a counterintuitive truth.

---

Two Teachers, Two Methods

Imagine you're a math teacher with struggling students. You have two strategies.

Strategy A: Traditional grading. Students attempt 100 problems; 12 are correct, 88 are wrong. You study both: the 12 correct solutions as positive examples, and the 88 wrong ones as negative examples, analyzing each error pattern so mistakes aren't repeated. This is the standard reinforcement learning approach. PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) both work this way: the model generates a set of rollouts, some correct (positive), some wrong (negative). Correct answers are rewarded, wrong ones punished, and the model improves through contrast.

Strategy B: The "excellent essays" method. Another teacher looks *only* at the 12 correct solutions, comparing them against each other: which solution is more elegant, more general, more robust? This teacher completely ignores the wrong answers, reasoning that the space of "error patterns" in the 88 failures is nearly infinite—calculation slips, conceptual confusion, typos, total misunderstanding—impossible to enumerate. But correct solutions *can* be compared with each other. By comparing within the set of correct answers, students naturally infer what "better correctness" looks like, while approaches absent from the good-solution set are implicitly disfavored—no explicit punishment needed.

This is the core idea of POPO (Positive-Only Policy Optimization).

---

Why Punishing Errors May Be Meaningless

RLVR (Reinforcement Learning with Verifiable Rewards) is the dominant paradigm for improving LLM reasoning: generate many answers, score them with a verifiable reward function (e.g., is the math answer correct, +1 or 0), and update the model. GRPO is its current star: for each question, generate a group of answers (e.g., 8), reward those above the group average, punish those below. It's simpler than PPO (no value network) and works surprisingly well.

But POPO's authors (Mingwei Xu and Hao Fang, both University of Washington) point to a problem GRPO doesn't fully address:

Under sparse binary rewards, punishing negative samples is nearly meaningless.

Consider a math competition problem. The model generates 8 answers: 2 correct (answer is 42), 6 wrong (some say 38, some 100, some gibberish). GRPO rewards the 2 and punishes the 6. But from the reward function's perspective, all 6 wrong answers are identical—a "41" (off by one) and a "banana" (totally irrelevant) receive the same zero punishment. As the paper puts it:

> "Negative rollouts may admit no gradation of failure severity."

And a second, more fatal issue:

> "The combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards."

The output space of math reasoning is effectively infinite. Punishing 6 sampled errors leaves infinitely many unseen errors the model may commit next—analogous to maintaining public safety by "listing all the bad people," an endless task. POPO's alternative: cultivate the good, making the standard for correct solutions ever clearer and higher, so anything below the standard is naturally eliminated.

---

*(The full analysis is published in multiple parts; continued in the replies below.)*