GRPO's Hidden Blind Spot: When Lucky Guesses Are Treated as Reasoning

Forum topic · ✨步子哥 · 2026-09-05

Summary

This post analyzes a recent paper exposing a systematic flaw in GRPO (Group Relative Policy Optimization), the standard RL algorithm behind DeepSeek-R1-style training. Because GRPO rewards any rollout that reaches the correct answer, guesses that land on the right answer receive the same advantage as genuine reasoning—a bias the authors call "spurious advantage." The problem is worst in bounded-answer tasks (multiple choice, classification), surprisingly affects ~56% of the open-ended MATH-7.5K dataset via small integers and fractions, and plagues multi-turn search agents where redundant paths can stumble onto correct answers. The proposed fix, SignBalance, decouples advantage magnitude from group statistics: the sign comes from the verifier, the magnitude is a global constant, and zero-mean force balance is restored with stop-gradient scaling—a one-line, zero-overhead change. Experiments show SignBalance matches GRPO on open-ended math while nearly doubling AMC accuracy, improving AIME, and beating Search-R1 by 1.8 points on QA benchmarks. The article argues outcome-based rewards generally conflate "correct answer" with "correct reasoning," a blind spot affecting most agent tasks with finite action spaces.

A Teacher's Dilemma

Imagine you're a math teacher with 100 students. You give a four-choice multiple-choice exam and collect 100 answers; 25 are correct. You give those 25 the same high score. In reality, 5 students actually computed the answer while 20 guessed—and you can't tell the difference, because you only looked at final answers, not the process. Worse, next semester you train new students on those 25 "success stories," and they learn not how to compute, but how to guess.

This isn't a teaching accident—it's what GRPO, the core RL algorithm behind current LLM post-training, is doing.

What GRPO Does

GRPO (Group Relative Policy Optimization) was proposed by DeepSeek in 2024 and used to train DeepSeek-R1. It's now the de facto standard for RL with verifiable rewards (RLVR).

The core logic is simple: for each question, sample a group of responses (say 8), score them with a rule-based verifier (correct=1, wrong=0), then normalize within the group—above-average answers get positive advantage, below-average get negative. Each rollout's advantage = (its reward − group mean) / group std.

Sounds reasonable. So where's the problem?

Guessing vs. Reasoning Are Indistinguishable

Consider a four-choice math question with 8 rollouts, 2 of which are correct. By the GRPO formula, both correct rollouts get the same high advantage—both trajectories are reinforced. But with four choices, pure guessing hits 25% of the time; across 8 rollouts, you'd expect ~2 lucky guesses. A substantial share of the "successful" trajectories you're rewarding may contain no reasoning at all.

The paper calls this spurious advantage: a gradient component from guessing, mixed indistinguishably with genuine reasoning advantage. And it's not limited to multiple choice.

Three High-Risk Scenarios

1. Bounded-answer tasks. Multiple choice, classification, finite-vocabulary QA—any limited answer space. A 25% blind-guess hit rate makes the spurious component non-negligible.

2. Hidden bounded subsets in open-ended data. Counterintuitively, MATH-7.5K—an open-ended math training set—contains many bounded answer forms:

| Answer form | Share | Blind-guess hit rate | Bounded? | |---|---|---|---| | Small integers [-10,10] | 24.41% | 4.76% | ✓ | | Mid integers [-100,100] | 22.99% | 0.50% | ✓ | | Simple fractions | 7.51% | 1.00% | ✓ | | Finite sets | 0.62% | 20.00% | ✓ | | Percentages | 0.42% | 0.99% | ✓ |

Bounded categories cover roughly 56% of MATH-7.5K. The paper even shows a model outputting the constant string "0" with no reasoning scores non-zero on MATH-7.5K.

3. Multi-turn search agents. In Search-R1-style agents, a trajectory can make up to 10 search calls over a huge action space. The same correct answer can be reached via countless search paths—many redundant, ineffective, or that wander off and accidentally hit the answer. GRPO's advantage only checks the final answer, so all lucky trajectories get equally high advantage. The agent learns "search more and hope," not "search efficiently."

SignBalance: A One-Line Fix

SignBalance decouples advantage magnitude from within-group correct/incorrect counts:

1. Class-wise normalization. Normalize correct and incorrect rollouts within their own subgroups, so the number of wrong rollouts doesn't affect correct rollouts' advantage magnitude. 2. Sign only, global magnitude. Replace the advantage magnitude with a global constant, keeping only the verifier's sign. This fully removes within-group dependence—spurious advantage no longer enters the gradient. Side effect: loss of batch-level zero-mean force balance hurts open-ended tasks. 3. Stop-gradient class-wise force balancing. Restore zero-mean balance with stop-gradient scaling factors per class; since no gradient flows through the scaling, the decoupling is preserved.

Final formula: sign from the verifier, magnitude as a global constant, force balance via stop-gradient class scaling. Zero parameters, PPO surrogate unchanged, no extra model or inference cost—essentially a one-line change.

Results

0.5B model (Qwen2.5-0.5B-Instruct, MATH-7.5K training):

| Method | Open-ended Avg | Bounded Avg | Overall Avg-8 | |---|---|---|---| | GRPO | 49.89 | 34.24 | 34.24 | | DAPO | 48.82 | 36.27 | 36.27 | | SignBalance | 49.66 | 36.61 | 36.61 |

SignBalance matches GRPO on open-ended math and leads across bounded tasks. AMC: 10.84 vs 6.02 (nearly double). AQuA: 35.43 vs 29.53 (+5.9).

3B model (Qwen2.5-3B-Base, 8 harder benchmarks): GRPO 42.80, DAPO 42.60, BNPO 43.18, SignBalance 43.78; AIME improves 7.7 → 8.5.

7B search agent (Qwen2.5-7B-Instruct, Search-R1 framework): Search-R1 36.00, StepSearch 36.44, SignBalance 37.80. Biggest gain on 2WikiMultiHopQA: 35.20 vs 27.58 (+7.6)—precisely where spurious advantage is worst.

Per-checkpoint accuracy curves on AQuA, SAT-Math, and MMLU-math diverge within ~50 steps and stay separated for hundreds of steps—systematic, not luck.

Deeper Insights

"Correct answer ≠ correct reasoning" is a systematic blind spot of outcome-based rewards. Any reward looking only at the final result has this problem whenever the answer space is finite. GRPO treats correctness as a scalar, but it's really a vector: correctness × reasoning quality. Collapsing dimensions echoes prior "scalar illusion" issues.
The evaluator-blindness pattern strikes again. GRPO's formula looks principled—group normalization, zero-mean balance—but it evaluates answer correctness, not reasoning quality. What an evaluator appears to measure and what it actually measures can differ by a key dimension.
Fix at a different level. SignBalance doesn't make the verifier smarter about reasoning quality; it removes the advantage formula's dependence on the contaminated dimension—solving the problem one level up rather than grinding at the same level.

Engineering Takeaways

1. Training multiple-choice, classification, or finite-output-space tasks? GRPO may be suboptimal; SignBalance is a near-zero-cost change. 2. Training search agents? Multi-turn trajectories + large action spaces = spurious-advantage hotspot; the largest gains appeared there. 3. Even "open-ended" math isn't safe—56% of MATH-7.5K is a hidden bounded subset. 4. SignBalance is a drop-in replacement: no architecture change, no data change, no PPO surrogate change, no inference overhead.

An Open Question

The paper validates SignBalance only on math reasoning and search agents; tool selection and code generation remain untested. But the authors speculate that choosing tools from a finite library is itself a bounded-answer problem subject to the same contamination. Whenever rewards depend only on final results and the result space is finite, spurious advantage exists—and most real agent tasks (tool calls, API selection, action planning) meet both conditions.

Paper Info

Title: Spurious Advantage Hidden in GRPO
Authors: Jiamian Wang (RIT), Samyadeep Basu, Koustava Goswami, Tong Yu, Zhiqiang Tao (Adobe Research / RIT)
arXiv: 2609.04063
Code: To be open-sourced
Date: 2026-09-03

---

*What's most admirable about this paper isn't the method itself—one line of math—but that it precisely identifies a blind spot everyone uses and no one noticed. Since 2024, GRPO variants (DAPO, Dr.GRPO, BNPO) have tweaked clipping ranges, normalization, and sampling, but nobody asked whether the advantage formula itself measures the right thing. It's like a crowd precisely calibrating a thermometer's markings without ever checking whether it measures temperature. Good research isn't always inventing new tools; sometimes it's pointing out the blind spots of old ones.*