A Teacher's Dilemma
Imagine you are a math teacher with 100 students. You give a four-choice multiple-choice question and collect 100 answers. 25 are correct, and you give all 25 the same high score.
In reality: 5 of them actually computed the answer, 20 guessed. You can't tell the difference — you only looked at the final answer, not the process. Worse, next semester you start training new students based on those 25 students' "success stories," and what they learn is not "how to compute" but "how to guess."
This isn't a teaching accident — it's what GRPO, the core RL algorithm for training today's large models, is doing.
What GRPO Does
Quick recap: GRPO (Group Relative Policy Optimization) was proposed by DeepSeek in 2024, used to train DeepSeek-R1, and is now the de facto standard for RLVR (RL with verifiable rewards) training.
The core logic is simple: for each question, the model generates a group of answers (say, 8 rollouts), a rule-based verifier scores them (correct = 1, wrong = 0), and rewards are normalized within the group — above-mean gets positive advantage, below-mean gets negative. The policy is updated to favor rollouts that perform well within the group.
Advantage of each rollout = (its reward − group mean reward) / group reward standard deviation.
Seems reasonable. So where's the problem?
"Guessed Right" vs. "Reasoned Right" Are Indistinguishable
Consider a four-choice math question. Out of 8 rollouts, 2 are correct. By the GRPO formula, those 2 correct rollouts receive the same high advantage — the model is encouraged to learn from these trajectories.
But for a four-choice question, pure guessing hits with 25% probability. Out of 8 rollouts, an expected 2 are pure guesses. In other words, a substantial fraction of the "successful trajectories" you're rewarding involve no reasoning at all — the rollout just happened to land on the correct option.
The paper calls this spurious advantage: a gradient component from guessing rather than reasoning, mixed with genuine reasoning advantage and learned equally.
Crucially, this is not limited to multiple choice.
Three Hot Zones
The paper identifies three scenarios where spurious advantage grows:
Scenario 1: Bounded-answer tasks. Multiple choice, classification, finite-vocabulary QA — any task with a limited answer space. With four choices, blind guessing hits 25% of the time — far too high for the spurious component to be negligible.
Scenario 2: Hidden bounded subsets in open-answer data. The most counterintuitive finding. MATH-7.5K is widely considered an "open answer" math training set, but the paper classified answers by syntactic form:
| Answer form | Share | Blind-guess hit rate | Bounded? | |---------|------|-----------|-------| | Small integers [-10,10] | 24.41% | 4.76% | ✓ | | Mid integers [-100,100] | 22.99% | 0.50% | ✓ | | Simple fractions | 7.51% | 1.00% | ✓ | | Finite sets | 0.62% | 20.00% | ✓ | | Percentages | 0.42% | 0.99% | ✓ |
Bounded classes cover roughly 56% of MATH-7.5K. You think you're training open reasoning, but over half the samples let the model guess numbers. The paper even ran a striking experiment: a model that does zero reasoning and always outputs the fixed string "0" scores non-zero on MATH-7.5K.
Scenario 3: Multi-turn search agents. Search-R1-style agents make up to 10 search calls per trajectory, with a huge action space. The same correct answer can be reached via countless search paths — many containing redundant searches, invalid queries, or complete detours that accidentally hit the right answer. GRPO's advantage formula only checks whether the final answer is correct, giving all "accidentally correct" trajectories the same high advantage. Agents learn "how to search more and get lucky" rather than "how to search efficiently."
SignBalance: A One-Line Fix
The fix, SignBalance, decouples advantage magnitude from the group's correct/incorrect ratio:
Step 1: Class-wise normalization. Correct and incorrect rollouts are normalized within their own subgroups, so the number of incorrect rollouts no longer affects the advantage magnitude of correct ones. Advantage still depends on group counts at this stage.
Step 2: Sign only; magnitude from a global scalar. Advantage magnitude is replaced by a global constant; only the verifier's sign is kept. This removes group-composition dependence entirely — spurious advantage no longer enters the gradient. Side effect: zero-mean force balance is lost, hurting performance on open-answer tasks.
Step 3: Class-wise force balance via stop-gradient. A stop-gradient scaling factor is applied per class (correct/incorrect) to restore zero-mean balance. Since no gradient flows through the scaling, Step 2's decoupling is preserved.
The final formula: signs from the verifier, magnitude a global constant, force balance restored via stop-gradient class scaling. Zero new parameters, PPO's surrogate objective unchanged, no extra model or inference cost. Essentially, one line changed.
The Numbers
0.5B model (Qwen2.5-0.5B-Instruct, trained on MATH-7.5K):
| Method | Open-answer Avg | Bounded-answer Avg | Overall Avg-8 | |------|------------|------------|---------| | GRPO | 49.89 | 34.24 | 34.24 | | DAPO | 48.82 | 36.27 | 36.27 | | SignBalance | 49.66 | 36.61 | 36.61 |
Parity with GRPO on open answers, a clean sweep on bounded answers. Most striking is AMC (competition multiple-choice math): SignBalance 10.84 vs GRPO 6.02 — nearly double. AQuA (five-choice math): 35.43 vs 29.53, +5.9 points.
3B model (Qwen2.5-3B-Base, harder 8-benchmark suite): SignBalance reaches 43.78 Avg-8, ahead of GRPO (42.80), DAPO (42.60), and BNPO (43.18); AIME rises from 7.7 to 8.5.
7B search agent (Qwen2.5-7B-Instruct, Search-R1 framework):
| Method | Avg-6 | |------|-------| | Search-R1 | 36.00 | | StepSearch | 36.44 | | SignBalance | 37.80 |
Highest average across six QA benchmarks, +1.8 over Search-R1. The biggest gain is on 2WikiMultiHopQA (the most complex multi-hop QA): 35.20 vs 27.58, +7.6 points — exactly the scenario with the worst spurious advantage: multi-turn search, large action space, redundant paths.
Stability Check
A natural objection: was one checkpoint just lucky? On AQuA, SAT-Math, and MMLU-math, per-checkpoint accuracy curves separate early (within ~50 steps), with SignBalance consistently above GRPO for hundreds of training steps. Not luck — a systematic difference.
Deeper Insights
"Correct answer ≠ correct reasoning" is a systematic blind spot of outcome-based rewards. Any reward that only looks at final results has this problem — whenever the answer space is bounded, guessing hits with non-zero probability. GRPO treats correctness as a scalar, but it's actually a vector: correctness × reasoning quality. Looking only at the correctness component, the other becomes a hidden contamination source.
Another case of the "evaluator blind-spot law." GRPO's formula looks sensible — group normalization, zero-mean balance — but it evaluates "is the answer right," not "is the reasoning right." Missing evaluation dimensions pollute the training signal. It is isomorphic to prior cases like LLM-judge omission blindness: what an evaluator appears to measure and what it actually measures can differ by a key dimension.
The training-side version of "judgment-gate decoupling." The verifier only judges answer correctness, not reasoning quality — two functions that should be coupled but are conflated in one scalar reward. SignBalance doesn't make the verifier smarter; it makes the advantage formula stop depending on the contaminated dimension — solving the problem at a different level rather than grinding away at the same one.
Engineering Implications
1. If you're training multiple-choice, classification, or finite-output tasks, GRPO may not be optimal. Spurious advantage is largest there, and SignBalance is a one-line, near-zero-cost change. 2. If you're training search agents, pay extra attention. Multi-turn trajectories + large action spaces = the worst zone; the paper's largest gain is on a 7B search agent. 3. Even "open answer" math isn't safe. MATH-7.5K has a 56% hidden bounded subset — half your "open reasoning" training is teaching the model to guess numbers. 4. SignBalance is drop-in. No architecture changes, no data changes, no PPO surrogate changes, no inference overhead — just change one line in your GRPO pipeline.
An Open Question
The paper honestly notes limitations: SignBalance is validated only on math reasoning and search agents; general scenarios (tool selection, code generation) are untested. But the conclusion offers an intriguing conjecture — when an agent must pick tools from a finite toolbox, that is itself a "bounded answer" problem, and GRPO likely suffers spurious advantage there too.
This points to a bigger issue: as long as the reward only sees final results and the result space is bounded, spurious advantage is everywhere. Most practical agent tasks — tool calls, API selection, action planning — meet both conditions. GRPO's blind spot may be broader than the paper shows.
Paper Info
- Title: Spurious Advantage Hidden in GRPO
- Authors: Jiamian Wang (RIT), Samyadeep Basu, Koustava Goswami, Tong Yu, Zhiqiang Tao (Adobe Research / RIT)
- arXiv: 2609.04063
- Code: Authors state it will be open-sourced
- Release date: 2026-09-03
*What I admire most about this paper isn't SignBalance itself — the method is one line — but how precisely it identifies a blind spot everyone uses and nobody notices. Since 2024, GRPO has spawned variants (DAPO, Dr.GRPO, BNPO) tweaking clip ranges, normalization, and sampling, but no one asked whether the advantage formula itself measures the right thing. It's like a crowd of people precisely calibrating a thermometer's markings without checking whether the thermometer measures temperature. Good research doesn't always invent new tools; sometimes it points out an old tool's blind spot.*