Breaking RLVR's Diversity Collapse: Why Correct but Uniform Answers Fall Short

> Paper: Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity > Authors: Anamika Lochab, Bolian Li, Ruqi Zhang > arXiv: 2605.00365 | 2026-04-29

1. The "AI Only Knows One Solution" Problem

Imagine asking a math question:

Problem: "Prove the Pythagorean theorem"

A human can respond with:

• An algebraic proof
• A geometric proof
• Similar triangles
• An area-based method
• Vectors
• ...

An AI trained with RLVR:

• Learns only one method
• Gives the same answer every time
• Even though the method is correct
• Lacks diversity

Consequence:

• Pass@1 (single-attempt accuracy) may be high
• But Pass@K (coverage over multiple attempts) is low
• The model knows only one solution path
• It may fail on problem variants

2. RLVR's Structural Flaw: Indifference to Diversity

The paper identifies a fundamental issue with RLVR (Reinforcement Learning with Verifiable Rewards):

Core finding: > Common RLVR objectives (such as GRPO) are *indifferent* to how probability is allocated among different correct solutions. This indifference allows probability mass to spontaneously concentrate on a narrow subset of correct solutions, causing diversity collapse.

Mechanism:

1. The cost of indifference

• The objective only cares whether the answer is correct
• It does not care how many distinct correct approaches exist

2. Self-reinforcing collapse

• Stochastic training dynamics
• + an indifference objective
• → probability concentrates spontaneously
• → a narrow subset monopolizes mass
• → diversity vanishes

3. A vicious cycle

• One solution gains a slight probability edge
• Training reinforces that trend
• Other solutions get squeezed out
• Only one remains

Analogy: A restaurant with many dishes, but customers only ever order the most popular one, so the chef gradually cooks only that dish—the menu becomes a single item.

3. Uniform-Correct Policy Optimization (UCPO)

Core idea: > Reward correctness, but also encourage a uniform distribution across correct solutions.

Technical approach:

1. Uniform reward — distribute reward evenly among correct solutions, favoring none 2. Breaking indifference — modify the objective with a diversity term that penalizes over-concentration 3. Preserving correctness — Pass@1 stays high while Pass@K improves 4. Theoretical guarantee — proof that collapse does not occur; diversity has a lower bound

4. Why Diversity Matters for Reasoning

Problems with low diversity:

• Fragility: one solution path means failure on edge cases and poor robustness
• No creativity: no exploration of new methods, only imitation
• Evaluation bias: high Pass@1 ≠ true understanding; it may just be memorization

Value of diversity:

• Robustness: multiple solutions enable cross-checking and fewer errors
• Exploration: thinking from different angles and discovering new approaches
• Genuine understanding: flexibility, not rote recall

5. A Feynman-Style Judgment

Feynman said:

> "If you can't explain it simply, you don't truly understand it."

Applied to AI reasoning:

> "If an AI can only answer a question one way, it hasn't truly understood—it has memorized the answer. True understanding means explaining the same concept from multiple angles and with multiple methods. UCPO teaches AI the 'diversity of thinking.'"

6. Takeaways

If you train reasoning models or use RLVR, ask yourself:

1. Is my model suffering from diversity collapse? 2. What does high Pass@1 but low Pass@K really mean? 3. Does my objective function encourage diversity? 4. Are multiple correct solutions treated equally?

UCPO reminds us: correctness is not the only goal—diversity matters just as much. When AI learns to be both "correct and diverse," it moves from answer machine to thinker. The best models of the future are not the fastest, but those that can think from the most angles.

*Original post and discussion: zhichai.net*