The Illusion of Robustness: Aggregate Accuracy Hides Per-Question Prediction Flips
Ask GPT-5.5 a multiple-choice question and it answers correctly. Now prepend 10,000 characters of gibberish—random pseudo-words with no relation to the question—and ask again. The model still gets it right. Aggregate accuracy barely moves, so it looks like the model is robust to irrelevant context.
But compare answers question by question, and a different picture emerges: while overall accuracy stays nearly the same, a substantial fraction of individual answers flip. Correct answers become wrong, wrong answers become right, and the two directions roughly cancel out—hiding the churn behind a stable headline number.
This is the phenomenon revealed by "The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context," a joint Georgia Tech and Stanford study published in July 2026.
The Experiment: Three Simple Steps
1. Pick 4 benchmarks: MMLU-Pro, GPQA-Diamond, Humanity's Last Exam, SimpleQA Verified 2. Sample 500 questions at random from each 3. Prepend a 10k-token "pseudo-word context" (nonsense words assembled from random characters, such as "qxlwbrt" and "zkpmnf"—semantically empty but matching English tokenization patterns) and re-ask
Models tested included gpt-5.5, gpt-5.4, gpt-4.1, DeepSeek-V4-Pro, Grok-4.20-NR, Mistral-Large-3, gpt-oss-120b, and gemini-3.1-fl.
Aggregate View: Almost No Change
Accuracy shifts after adding pseudo-word context:
- gpt-5.5: ±0.9%
- gpt-5.4: ±2.1%
- Other models: mostly within ±2%
- INS (Instability): the fraction of questions whose answers flipped
- WTD (Word-Token Distance): the magnitude of answer changes
- Some questions that were answered correctly degrade with context
- Some questions that were answered wrong improve with context
- Test the same question repeatedly under multiple context conditions
- Report answer-flip rates (INS), not just mean accuracy
- Examine the distribution of flips, not just the mean
This level of fluctuation is indistinguishable from sampling noise. Judged on aggregate accuracy alone, the conclusion would be "the model is robust to irrelevant context"—which is exactly what most existing evaluations conclude.
Per-Question View: Answers Flip
The authors went further and compared answers question by question: same question, with vs. without context—did the answer change?
Even with entirely meaningless pseudo-words, a considerable proportion of answers flipped. Some previously correct answers degraded; some previously wrong ones improved. Both directions occur in roughly equal numbers, so they cancel in the aggregate.
Two metrics quantify the phenomenon:
Key finding: INS is nonzero across all models and all benchmarks. Even the strongest model, gpt-5.5, changes some answers in the face of pure random pseudo-words.
Why Does Gibberish Affect Answers?
This is the most counterintuitive part. The authors offer no definitive answer but discuss candidate mechanisms:
1. Attention distribution shifts. Attention assigns weight to all input tokens. Even semantically empty pseudo-words consume attention budget, diluting attention to the question itself—like someone rustling paper beside you during an exam. 2. Position encoding effects. Adding context changes the question's positional encoding; the model may treat "the word at position 100" and "the word at position 10100" differently even with identical content. 3. Normalization effects. Long context alters softmax distributions; tens of thousands of tiny attention weights can accumulate enough to change the relative size of final logits. 4. Training distribution mismatch. Training inputs rarely contain pure random pseudo-words; such out-of-distribution input may yield unstable behavior.
Flips Are Bidirectional
One of the paper's most important insights: flips are not one-directional degradation.
The counts are roughly equal, so overall accuracy barely moves. The "improvements" may arise because even gibberish perturbs the model's reasoning path, occasionally bypassing a habitual error mode—like being interrupted and taking a different line of thought. But this is not a good thing: the improvement is random, uncontrollable, and unexplainable. You cannot depend on "maybe gibberish will help" in critical applications.
Flips Are Model-Specific
The questions that flip for different models barely overlap. A question that flips for gpt-5.5 may not flip for gpt-5.4 or for models from other labs. Instability is not an intrinsic property of the question but a model-specific weak spot.
Practical implication: multi-model ensembling can substantially reduce instability, since independent flips can be cancelled by majority voting. The caveat is that models must be sufficiently independent—a premise that is getting harder to satisfy given ongoing model convergence trends.
What Modulates Instability?
The authors systematically tested several variables:
1. Context type: pseudo-words < real but irrelevant text < semi-relevant text. The more "meaningful" the context, the more flips it induces. 2. Context length: from 1k to 10k tokens, longer context causes more flips, though growth saturates beyond some length rather than increasing linearly. 3. Test-time compute: models with reasoning mode enabled show significantly lower instability; reasoning tokens give the model a chance to "re-examine," partially offsetting contextual interference. 4. Model generation: newer models are more stable overall, but not immune. gpt-5.5's INS is lower than gpt-4.1's, yet still nonzero.
Why It Matters
1. A blind spot in evaluation. Existing benchmarks almost universally report aggregate accuracy. If the headline number doesn't change, the model is deemed "robust." But this misses violent per-question instability. A model that looks 99% accurate in aggregate may flip unpredictably on 1% of questions—and that 1% may be exactly where the critical application lives.
2. Tail risk in deployment. Real deployments almost always involve long context—codebases, conversation history, retrieval results. If every long context carries a small probability of flipping an answer, flips become nearly inevitable across thousands of calls—and you can't predict which call will be hit.
3. Cascading effects in agent systems. Agents reason over multiple steps. A small per-step flip probability compounds exponentially: one wrong step poisons everything downstream. This is why agent systems are more fragile than single-turn QA—"small hallucinations" at each step accumulate into "large disasters."
The Authors' Recommendation: Per-Question Evaluation
The authors call on the community to adopt per-question stability evaluation as part of standard evaluation pipelines:
A Deeper Reflection
The paper evokes Simpson's paradox in statistics: aggregate trends and subgroup trends can point in opposite directions. Here, stable aggregate accuracy masks violent per-question volatility.
The implication for AI safety runs deeper. We increasingly rely on aggregate metrics to assess model capability, but aggregates inherently obscure distributional detail. A model with "excellent average performance" may harbor fatal weaknesses on specific inputs—weaknesses that only surface under per-question analysis.
Robustness is not a number; it is a distribution. Look only at the number, and you fall for an illusion.
---
Paper: https://arxiv.org/abs/2607.12963 HTML full text: https://arxiv.org/html/2607.12963 Code and data: https://github.com/SALT-NLP/illusion-of-robustness