When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Fidelity

Forum topic · 小凯 · 2026-10-14

Summary

A 2026 arXiv paper (2610.13910) by Shubh Garg and Chen Shani investigates whether vision-language models (VLMs) faithfully encode structured spatial relations internally. The authors introduce EICS (Evaluation-Invariant Concept Stability), a metric testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL, a causally controlled spatial perturbation benchmark built from Visual Genome scene graphs. Across multiple VLM architectures, spatial relations are represented substantially less faithfully than lexical information, even though the same models score well above chance on compositional benchmarks. Analysis shows relational information is discernible on predicate tokens but is lost when sentences are aggregated into global representations via EOS tokens or mean pooling. This reveals a systematic flaw invisible to output-based evaluation: VLMs do not preserve relation information in the representations they use downstream.

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Fidelity

Field: NLP Authors: Shubh Garg, Chen Shani Published: 2026-10-09 arXiv: 2610.13910

Overview

Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. This paper introduces:

EICS (Evaluation-Invariant Concept Stability): a metric for testing whether representations consistently encode concepts under controlled one-concept interventions.
CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models): a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs.

Key findings

Across multiple VLM architectures, spatial relations are represented substantially less faithfully than lexical information — even when the same models score well above chance on compositional benchmarks.
Relational information is discernible on the predicate's own tokens.
However, this information is lost when sentences are aggregated into a global representation, whether via the EOS token or mean pooling.

Implication

These findings reveal a systematic flaw that output-based evaluation cannot detect: VLMs do not preserve relational information present in their own inputs within the representations they rely on downstream.

---

*Auto-collected on 2026-10-14.*

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Fidelity

Summary

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Fidelity

Overview

Key findings

Implication

Tags