Mirage: When Multimodal AI 'Sees' Images That Were Never Uploaded

Forum topic · 小凯 · 2026-05-28

Summary

A March 2026 Stanford paper by Fei-Fei Li's team, 'Mirage: The Illusion of Visual Understanding,' exposes a fundamental flaw in how we evaluate multimodal AI. When tested on visual benchmarks without supplying any images, frontier models such as GPT-5.2 generated detailed descriptions of non-existent pictures and answered anyway, with a mirage rate of 93.5%. A 3-billion-parameter text-only model called Super-Guesser reached roughly 75% accuracy on medical vision benchmarks, beating both top multimodal models (60-65%) and radiologists (55-60%). Up to 75% of items in benchmarks like MicroVQA, MedXpertQA-MM, and MMMU-Pro can be answered correctly without images, relying on question wording and option statistics. The team proposes B-Clean, a filtering method that strips non-visual items and reorders leaderboards accordingly. The findings demand image-presence detection, private test sets, and image-vs-no-image accuracy gaps as standard practice.

What is Mirage?

Stanford's Fei-Fei Li team published "Mirage: The Illusion of Visual Understanding" in March 2026. Unlike hallucination, where an AI sees the image but mis-describes details, mirage occurs when the model never sees the image at all, fabricates a detailed description, and reasons confidently from the fabrication.

When GPT-5.2, Gemini 2.5/3 Pro, Claude Opus/Sonnet 4.5, and others were asked visual questions with no image supplied, 93.5% of GPT-5.2's responses still described an image that did not exist.

A Text-Only Model Beats Radiologists

The team fine-tuned a 30-billion-parameter text-only model (Qwen2.5-3B) on 570,000 medical visual questions with all images removed, called Super-Guesser. Results on a private test set:

| Evaluator | Accuracy | |-----------|----------| | Super-Guesser (3B text-only) | ~75% | | Top multimodal models (average) | ~60-65% | | Radiologists (average) | ~55-60% |

The text-only model outscored human specialists by roughly 10 points and generated reasoning traces indistinguishable from real visual reasoning.

Two Modes: Mirage vs Guess

Models use two distinct strategies when no image is available:

Mirage-mode: Pretend the image exists, generate a fake description, reason from it. Higher accuracy.
Guess-mode: Acknowledge the missing image and guess from priors. Lower accuracy.

The same model on the same question can swing 10-15% in accuracy based purely on prompt phrasing. Conventional "blind-guessing controls" systematically underestimate benchmark fragility.

How Dangerous Is Mirage in Medicine?

Gemini-3-Pro's most common fabricated diagnoses in 200 repeated trials:

ECG: acute STEMI (immediate surgery indicated)
Dermatology: malignant melanoma (urgent excision)
Pathology slides: various carcinomas

A silent failure scenario: a user uploads a skin photo, the network drops the file, the model produces a detailed melanoma diagnosis without flagging the missing image, and the patient undergoes an unnecessary procedure.

B-Clean: Removing 75% of "Fake Visual Items"

The team's B-Clean filter runs each benchmark in no-image mode. Items answered correctly without an image are flagged as non-visual and removed:

| Benchmark | Original | Retained | Removed | |-----------|----------|----------|---------| | MicroVQA | 1,042 | 240 | 77% | | MedXpertQA-MM | 2,000 | 514 | 74% | | MMMU-Pro | 1,730 | 428 | 75% |

After cleaning, leaderboards shift dramatically. On MicroVQA, Gemini-3-Pro drops from 68.8% to 23.2% and GPT-5.1 from 61.5% to 15.4%.

Why This Matters

1. Evaluation: Most existing multimodal benchmarks test text statistics, not vision. 2. Products: Any vision-input AI needs modality-presence detection; a 200 OK response does not prove the image reached the model. 3. Cognition: Current models navigate the statistical space of visual questions rather than truly understand images.

The Authors' Three Recommendations

1. Every multimodal evaluation must include a no-image control for every input modality. 2. Benchmarks must remain private, with rotating items, to resist pretraining contamination. 3. Report the image-vs-no-image accuracy gap, not just absolute accuracy, as the true measure of visual understanding.

References

Asadi et al. (2026). *MIRAGE: The Illusion of Visual Understanding*. arXiv:2603.21687. https://arxiv.org/abs/2603.21687
Related paper: O'Sullivan et al. (2026). *MARCUS: an agentic, multimodal vision-language model for cardiac diagnosis and management*. arXiv:2603.22179
Benchmarks: VQA-RAD, MicroVQA, MedXpertQA-MM, MMMU-Pro, ReXVQA