Breaking RLVR's Diversity Collapse: Why Correct but Homogeneous Answers Fall Short — An Introduction to UCPO

> Paper: Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity > Authors: Anamika Lochab, Bolian Li, Ruqi Zhang > arXiv: 2605.00365 | 2026-04-29

1. The "AI Only Knows One Solution" Problem

Ask an AI to prove the Pythagorean theorem. Humans can approach it via algebra, geometry, similar triangles, area methods, or vectors. After RLVR training, however, a model may produce the same single solution every time — correct, but lacking diversity.

Consequences:

• Pass@1 (single-attempt accuracy) can remain high
• Pass@K (coverage over multiple attempts) drops
• The model effectively knows one solution path, and may fail on problem variants

2. RLVR's Structural Flaw: Indifference to Diversity

The paper's core finding:

> Common RLVR objectives (such as GRPO) are indifferent to how probability is distributed among different correct solutions. This indifference allows probability mass to spontaneously concentrate on a few correct solutions, causing diversity collapse.

Mechanism:

1. Cost of indifference — the objective only cares whether an answer is correct, not how many correct ways exist (like grading exams only on right/wrong). 2. Self-reinforcing collapse — stochastic training dynamics + an indifferent objective → probability mass concentrates → a narrow subset monopolizes → diversity vanishes. 3. Vicious cycle — a solution gaining slightly higher probability gets reinforced by training, other solutions get squeezed out, until only one remains.

An analogy: a restaurant with many dishes where customers only order the most popular one — eventually the chef cooks only that dish, and the menu loses all variety.

3. Uniform-Correct Policy Optimization (UCPO)

The proposed solution:

> Reward correctness, and encourage a uniform distribution over correct solutions.

Key components:

1. Uniform rewards — rewards are spread evenly across correct solutions, favoring none. 2. Breaking indifference — the objective is modified with a diversity term that penalizes over-concentration. 3. Preserving correctness — Pass@1 stays high while Pass@K improves. 4. Theoretical guarantees — proofs that diversity does not collapse and maintains a lower bound.

4. Why Diversity Matters for Reasoning

Problems with low diversity:

• Fragility: one solution path means failure on edge cases; no robustness.
• No creativity: no exploration of new methods, only imitation.
• Misleading evaluation: high Pass@1 ≠ real understanding; it may be memorized answers rather than reasoning.

Value of diversity:

• Robustness: multiple solutions enable cross-checking and fewer errors.
• Exploration: thinking from different angles, discovering new approaches, creative reasoning.
• Genuine understanding: the ability to explain from multiple perspectives, not rote memorization.

5. A Feynman-Style Take: Understanding = Explaining from Multiple Angles

Feynman said: *"If you can't explain it simply, you don't understand it well enough."*

Applied to AI reasoning: if a model can only answer a question one way, it hasn't truly understood — it has memorized an answer. True understanding means explaining the same concept from multiple angles and via multiple methods. UCPO teaches models the *diversity of thinking*.

This mirrors education itself: the goal is not to memorize one answer, but to understand many methods and apply them flexibly.

6. Takeaways

If you train reasoning models or use RLVR, ask:

1. Does my model suffer from diversity collapse? 2. What does high Pass@1 with low Pass@K mean? 3. Does my objective function encourage diversity? 4. Are multiple correct solutions treated equally?

UCPO reminds us: correctness is not the only goal — diversity matters just as much. A model that learns to be both "correct and diverse" moves from answer machine to thinker. In the garden of intelligence, diversity is the most beautiful flower.