The Illusion of Robustness: Stable Aggregate Accuracy Hides Per-Question Prediction Flips
Imagine testing GPT-5.5 on a multiple-choice question. The question is clear, the answer unambiguous, and the model gets it right. Now prepend 10,000 characters of pure gibberish—random pseudo-words with no relation to the question—and ask again. The model still answers correctly. You relax: the model seems robust to irrelevant context.
But what if you ran this test on every question and compared answers one by one?
Aggregate accuracy barely changes—yet for individual questions, a substantial fraction of answers flip. Questions answered correctly become wrong, and vice versa. The two directions roughly cancel out, so the headline numbers look calm.
This is the phenomenon revealed by "The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context," a July 2026 study jointly published by Georgia Tech and Stanford. The authors call it the "illusion of robustness."
The Experimental Design: Just Three Steps
1. Select 4 benchmarks: MMLU-Pro, GPQA-Diamond, Humanity's Last Exam, SimpleQA Verified 2. Randomly sample 500 questions per benchmark 3. Prepend 10,000 tokens of "pseudo-word context" (nonsense words built from random characters) and re-ask
The pseudo-words are formed by randomly combining characters into word-like strings such as "qxlwbrt" and "zkpmnf"—semantically empty but matching English tokenization patterns.
Models tested include: gpt-5.5, gpt-5.4, gpt-4.1, DeepSeek-V4-Pro, Grok-4.20-NR, Mistral-Large-3, gpt-oss-120b, and gemini-3.1-fl.
Aggregate View: Almost No Change
After adding pseudo-word context, accuracy shifts:
- gpt-5.5: ±0.9%
- gpt-5.4: ±2.1%
- Other models: mostly within ±2%
- INS (Instability): the fraction of questions whose answers flip
- WTD (Word-Token Distance): the magnitude of answer changes
- Some questions answered correctly become wrong (degradation)
- Some questions answered incorrectly become right (improvement)
- Test each question multiple times under multiple context conditions
- Report answer flip rates (INS), not just average accuracy
- Examine the distribution of flips, not just the mean
This level of variation is comparable to sampling noise. Looking only at aggregate accuracy, the conclusion would be "models are robust to irrelevant context."
This is also what most existing evaluations conclude.
Per-Question View: Answers Flip
But the authors didn't stop at summary numbers. They compared answers question by question: same question, with context vs. without—did the answer change?
The result is striking:
Even with completely meaningless pseudo-word context, a considerable fraction of answers flip. Some previously correct answers become wrong; some previously wrong ones become right. Both directions occur in roughly equal numbers, canceling out at the aggregate level.
The authors define two metrics:
INS tells you "how many questions flipped"; WTD tells you "how far they flipped."
Key finding: INS is nonzero across all models and all benchmarks. Even the strongest model, gpt-5.5, changes answers on a small fraction of questions when faced with pure random pseudo-words.
Why Does "Gibberish" Affect Answers?
This is the most counterintuitive part. Pseudo-words carry no semantics—why would prepending gibberish change an answer?
The authors offer no definitive answer but discuss several possible mechanisms:
1. Attention distribution shifts. Transformers allocate attention weights across all input tokens. Even without semantics, pseudo-words consume attention budget, reducing attention to the actual question. Like someone shuffling papers beside you during an exam—the noise is meaningless, but it distracts you.
2. Positional encoding effects. With context prepended, the question's positional encodings change. The model may treat "the word at position 100" differently from "the same word at position 10,100."
3. Normalization effects. Long context alters softmax distributions. Even if each pseudo-word gets a tiny attention weight, tens of thousands of small weights can shift the relative magnitude of final logits.
4. Training distribution mismatch. Models mostly see meaningful long contexts during training. Pure random pseudo-words are out-of-distribution input, and behavior may be unstable.
The Crucial Finding: Flips Are Bidirectional
This is one of the paper's most important insights.
Answer flips are not one-directional—not simply "context makes things worse." They go both ways:
The two categories are roughly equal in number, so aggregate accuracy barely moves.
Why would performance "improve"? One explanation: for certain hard questions, extra context—even gibberish—changes the model's reasoning path, accidentally bypassing its original error mode. Like a person stuck on one line of thought who, once interrupted, finds another path.
But this is not a good thing. The "improvement" is random, uncontrollable, and unexplainable. You cannot know which questions will improve and which will degrade. In critical applications, you cannot rely on "maybe gibberish will help."
Flips Are Model-Specific
The authors also found: the questions that flip barely overlap across models.
A question that flips for gpt-5.5 may not flip for gpt-5.4; a question that flips for one model may not flip for another. Instability is not a property of the question itself, but a model-specific weak point.
Practical implication: multi-model ensembles can significantly reduce this instability. If models flip independently, majority voting cancels most errors—provided the models are sufficiently independent. Given the model-convergence trends discussed elsewhere, that premise is increasingly hard to satisfy.
What Modulates Instability?
The authors systematically tested several variables:
1. Context type: pseudo-words < real but irrelevant text < semi-relevant text. The more context "resembles meaningful text," the more flips it induces.
2. Context length: from 1k to 10k tokens, longer context causes more flips—but growth is not linear; it saturates beyond some length.
3. Test-time compute: models with reasoning mode enabled show significantly lower instability. Reasoning tokens give the model a chance to "re-examine," partially offsetting contextual interference.
4. Model generation: newer models are more stable overall but not immune. gpt-5.5's INS is lower than gpt-4.1's, but still nonzero.
Why This Matters
The paper's warning operates on three levels:
1. A blind spot in evaluation. Existing benchmarks almost universally report aggregate accuracy. If the summary number doesn't change, the model is deemed "robust." But this judgment misses violent per-question instability. A model that looks 99% accurate in aggregate may flip unpredictably on 1% of questions—and that 1% may be exactly the critical application scenario.
2. Tail risk in deployment. In real settings, models almost always face long contexts—codebases, conversation histories, retrieved documents. If every long context carries a small probability of flipping an answer, then across thousands of calls, flips become nearly inevitable—and you don't know which call will hit.
3. Cascading effects in agent systems. Agents reason over multiple steps. A small per-step flip probability compounds exponentially over a trajectory: one wrong step derails everything downstream. This is why agent systems are more fragile than single-turn QA—"small hallucinations" at each step accumulate into "large disasters."
The Authors' Recommendation: Per-Question Evaluation
The authors call on the community to adopt per-question stability evaluation as part of standard assessment pipelines:
A Deeper Reflection
This paper evokes Simpson's paradox from statistics: aggregate trends and subgroup trends can point in opposite directions. Here too—stable aggregate accuracy masks violent per-question fluctuations.
The phenomenon carries deeper implications for AI safety research. We increasingly rely on aggregate metrics to judge model capability, but such metrics inherently obscure distributional detail. A model with "excellent average performance" may harbor fatal weaknesses on specific inputs—weaknesses that only surface under per-question analysis.
Robustness is not a number; it is a distribution. Look only at the number, and you fall for an illusion.
---
Paper: https://arxiv.org/abs/2607.12963 HTML full text: https://arxiv.org/html/2607.12963 Code and data: https://github.com/SALT-NLP/illusion-of-robustness