[论文] When VLMs Destroy What They Know: Spatial Representations Fall Below L...

论文概要

研究领域: NLP 作者: Shubh Garg, Chen Shani 发布时间: 2026-10-09 arXiv: 2610.13910

中文摘要

视觉-语言模型（VLM）越来越多地分析和推理图像，但其内部表征是否忠实地编码了此类推理所需的结构化关系仍不清楚。我们引入了 EICS（评估不变概念稳定性）——一种在受控的单概念干预下测试表征是否一致编码概念的指标，以及 CASP-VL——一个基于 Visual Genome 场景图构建的因果受控空间扰动基准。在多种 VLM 架构上，我们发现空间关系的表征忠实度显著低于词汇信息，即使这些模型在组合式基准上得分远高于随机水平。进一步分析表明，关系信息在谓词自身的 token 上可辨识，但当句子被聚合为全局表征时（无论是通过 EOS token 还是均值池化）就会丢失。这些发现揭示了一种基于输出的评估无法发现的系统性缺陷：VLM 并未在其下游使用的表征中保留自身输入中存在的关系信息。

原文摘要

Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models), a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across multiple VLM architectures, we find that spatial relations are represented substantially less faithfully than lexical information, even when the same models score well above chance on compositional benchmarks. Further analy...

#论文 #arXiv #NLP #小凯