In March 2026, Stanford's Fei-Fei Li team published a paper titled *MIRAGE: The Illusion of Visual Understanding*, with three findings that challenge our trust in multimodal AI.
The most extreme case: a 3-billion-parameter, text-only model took first place on a chest X-ray question-answering benchmark — without ever seeing a single X-ray.
Mirage vs. Hallucination
- Hallucination: the model sees the image but gets details wrong (e.g., describing a cat as a dog). The input is real; the output is biased.
- Mirage: the model never sees the image at all, yet generates a detailed description of this nonexistent image and reasons on top of it. The input doesn't exist; the entire cognitive frame is fabricated.
- Parameters: 3 billion (Qwen2.5-3B)
- Training data: 570k medical visual questions, all images removed
- Released September 2024 (9 months before the test benchmarks, ruling out contamination)
- Mirage mode: asked a visual question directly with no mention that the image is missing, the model assumes the image exists, fabricates a description, and reasons from it. Accuracy is high.
- Guess mode: explicitly told "no image, guess the best answer," the model becomes conservative and relies on priors and answer distributions. Accuracy is low.
- Asadi et al. (2026). MIRAGE: The Illusion of Visual Understanding. arXiv:2603.21687. https://arxiv.org/abs/2603.21687
- Related: O'Sullivan et al. (2026). MARCUS: an agentic, multimodal vision-language model for cardiac diagnosis and management. arXiv:2603.22179
- Benchmarks tested: VQA-RAD, MicroVQA, MedXpertQA-MM, MMMU-Pro, ReXVQA
The team tested all frontier models — GPT-4.1/5/5.1/5.2, Gemini 2.5/3 Pro, Claude Opus/Sonnet 4.5 — asking visual questions without providing any images.
Result: every model "saw" nonexistent images. GPT-5.2's mirage rate reached 93.5%.
Winning Without Images
Accuracy without images was remarkably close to accuracy with images:
| Benchmark | Domain | No-image accuracy as % of full accuracy | |---|---|---| | VQA-RAD (radiology) | Medical | ~95% | | MicroVQA (microscopy) | Medical | ~85% | | MedXpertQA-MM (general medicine) | Medical | ~90% |
The team then trained Super-Guesser, a text-only model:
On a private test set:
| Evaluator | Accuracy | |---|---| | Super-Guesser (3B text-only) | ~75% | | Frontier multimodal models (average) | ~60-65% | | Radiologists (average) | ~55-60% |
The text-only model also produced reasoning traces "indistinguishable from genuine visual reasoning."
Mirage Mode vs. Guess Mode
The same model on the same question can differ by 10-15% depending on the prompt. GPT-5.1, for example: MicroVQA ~55% vs ~45%; MedXpertQA-MM ~70% vs ~55%; MMMU-Pro ~65% vs ~50%. In 18 of 23 MMMU-Pro categories, mirage mode beat guess mode. This means conventional "blind-guess controls" systematically underestimate how vulnerable benchmarks are.
The Medical Danger
In 200 repeated runs with Gemini-3-Pro, the most frequent phantom diagnoses were:
| Modality | Most common phantom diagnosis | Urgency | |---|---|---| | ECG | Acute STEMI | Immediate intervention | | Dermatology | Malignant melanoma | Emergency excision | | Pathology slides | Various cancers | Cancer treatment |
Worst of all, the failure is silent: the model never says "I didn't see an image" — it just delivers a confident diagnosis. Users cannot distinguish real image analysis from fabrication.
B-Clean: Cleaning Out Pseudo-Visual Questions
The proposed fix, B-Clean, runs benchmarks in no-image mode; questions answerable without images are removed as non-visual.
| Benchmark | Original | Damaged | Retained | Removed | |---|---|---|---|---| | MicroVQA | 1,042 | 802 | 240 | 77% | | MedXpertQA-MM | 2,000 | 1,486 | 514 | 74% | | MMMU-Pro | 1,730 | 1,302 | 428 | 75% |
After cleaning, rankings reshuffled dramatically. On MicroVQA: Gemini-3-Pro fell from 68.8% to 23.2% (-45.6); GPT-5.1 fell from 61.5% to 15.4% (-46.1).
Why It Matters
1. Evaluation: current multimodal benchmarks are unreliable — ~75% of questions test text statistics, not vision. 2. Product: any AI app with visual input needs missing-modality detection. A 200 API response doesn't mean the image actually arrived; the model won't tell you. 3. Conceptual: models aren't "understanding images" — they're "navigating the statistical space of visual questions." The image is a trigger, not the fuel.
The Authors' Three Recommendations
1. Every multimodal evaluation must include no-image controls, systematically disabling each input modality. 2. Benchmarks must be kept private, with rotating question sets, since public benchmarks are absorbed into the next pretraining run. 3. Measure the with-image minus without-image gap — that difference, not absolute accuracy, is the true measure of visual understanding.