VLM Spatial Awareness Hallucinations: When Models Get the Right Answer Without Understanding

A recent paper from Tel Aviv University asks a deceptively philosophical question: when CLIP correctly answers "Is the cup on the table?", does it actually understand the spatial relation "on", or does it merely exploit the statistical co-occurrence of the words "cup" and "table"? The authors' experiments deliver an unsettling answer.

A Simple Controlled Experiment

Show the model two images—one where a cup is on a table, one where it is beneath the table—and swap "on" for "beneath" in the text while controlling everything else. If the model truly encodes spatial relations, its internal vectors should shift in a clear, directionally opposite way.

This motivates the paper's EICS (Evaluation-Invariant Concept Stability) metric: change exactly one concept and check whether the model's internal representations track the change. As a baseline, they use a naive bag-of-words (BoW) model that ignores word order entirely. The logic: if even BoW can distinguish information that a deep VLM cannot, something is seriously wrong.

The Shocking Numbers

| Concept type | BoW baseline | CLIP L12 | Retention | |---------|---------|----------|-------| | Color (red/blue/...) | +0.332 | +0.312 | 94% | | Spatial relations (on/beneath/...) | +0.372 | +0.070 | 19% |

Color is almost fully preserved (94% of the BoW signal), but spatial relations survive at only 19%—CLIP discards 81% of spatial information when aggregating its representation.

Even more striking: SigLIP achieves the highest relation accuracy on SugarCrepe (70.6%) among three compared models, yet has the lowest spatial EICS (+0.038), with a confidence interval lower bound near zero. The top-scoring student has the worst spatial understanding—this is not irony, it is an evaluation blind spot.

Where Is the Information Lost?

The authors localize the loss by comparing two representations: the pooled [EOS] sentence embedding versus the hidden states at the predicate token positions (where "on"/"beneath" sit).

At the predicate token itself, spatial relations are identifiable. But once pooled into a sentence-level representation, the signal is washed out—analogous to information getting diluted through layers of organizational reporting. Mean pooling barely helps: spatial EICS rises only from +0.070 to +0.093, far below BoW's +0.372. No pooling strategy preserves spatial information.

Can the Image Side Compensate?

One might hope the visual encoder provides the missing evidence. Testing image-caption similarity before and after intervention shows that visual grounding helps for color interventions but is nearly useless for spatial ones: after a spatial swap, images prefer the original caption only 34% of the time—slightly below chance. CLIP's image encoder itself fails to encode "on vs. beneath". Vision and language are doubly blind to space.

Why It Matters

The paper's deep contribution is revealing three levels of decoupling:

1. Behavioral correctness ≠ representational faithfulness. Models can score 70% on spatial benchmarks while their representations encode no spatial relations—relying on lexical co-occurrence statistics instead. 2. Benchmark accuracy ≠ representation quality. SigLIP is the most accurate on SugarCrepe and the weakest on spatial EICS. The two metrics are not merely inconsistent but inversely correlated, meaning current evaluation systems can systematically mislead us. 3. Local encoding ≠ global availability. Spatial relations are readable at predicate tokens but erased during aggregation.

Implications for AI Safety

The paper fits into a lineage of "evaluation blind spot" research:

• Citation faithfulness (Cited but Not Consulted): models cite legal statutes whose removal doesn't change their verdicts
• The certainty trap (TT = C·(A - φ/2)): 100% confidence with only 51% correctness
• Capability-anchoring traps (METR): globally predictive metrics failing in local regimes

Now add the representational faithfulness trap: getting the answer right does not mean the internal representation encodes the relevant information. Counterfactual audits—perturb one concept and check whether representations change—are the correct methodology, just as swapping cited statutes audits citation faithfulness.

The right question is not "How accurate is the VLM at spatial understanding?" but "How much spatial information survives at the representation level?" These two answers can be complete opposites.

A Chilling Analogy

Imagine a translator who flawlessly renders "the cup is on the table" into ten languages. Swap the image so the cup is actually under the table, and the translation doesn't change—because the translator never processed the spatial relation, only the fixed pairing of "cup" and "table". CLIP is that translator.

Deployed in autonomous driving, robot navigation, or medical imaging—domains requiring genuine spatial reasoning—the gap between "answering correctly" and "understanding space" becomes the gap between success and accident. The paper provides a ruler (EICS) and a testbed (CASP-VL) to quantify that gap. The answer: shallower than a bag-of-words model.

---

Paper: When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

Code: github.com/Shubhf/CASP-VL

Authors: Shubh Garg, Chen Shani (Tel Aviv University)