[论文] [论文] Do vision models know that three is three?

论文概要

研究领域: CV
作者: Ankit Sonthalia, Arnas Uselis, Seong Joon Oh
发布时间: 2026-10-26
arXiv: 2610.35857

中文摘要

视觉模型是否以同样的方式编码「2→3」这一转变——无论它出现在手写数字还是手势中？它们是否理解「干净→肮脏」是同一个变换，无论施加在厨房还是地毯上？线性表征假说（LRH）对语言模型给出了肯定回答：概念变化对应嵌入空间中的方向。但在视觉中，同一概念可能出现在视觉上毫不相关、极少共现的 regime 中——数字图像与手的图像在像素空间几乎毫无共同之处。为弄清视觉模型是否仍编码底层抽象概念，我们研究了 46 个视觉编码器（跨架构、训练目标与规模）在 10 个概念族（数量、年龄、物体身份、MIT-States 属性等）上的表现。每个 regime 内部，几乎所有模型都存在统计显著的概念方向；但两个不同 regime 并不总是共享同一方向——若干模型为同一概念转变学习了多个线性结构，分散在不同 regime 中。我们将此称为「碎片化线性」：强的局部线性与近乎缺失的全局线性共存。不过碎片化并非不可改变：局部线性越强的模型越能逐步摆脱它（局部-全局线性相关 τ∈[0.38,0.89]）；全局线性确实出现之处，概念方向可作为线性探针跨 regime 迁移。

原文摘要

Do vision models encode the transition \(2{\to}3\) the same way whether it appears in handwritten digits or in hand gestures, and do they "understand" that clean\({\to}\)dirty is the same transformation regardless of whether it is applied to a kitchen or a carpet? The linear representation hypothesis (LRH) answers this for language models: concept changes correspond to directions in (language) embedding space. In vision, the same concept can manifest in visually unrelated "regimes" that rarely co-occur: images of digits and hands do not share much in common in pixel space. To understand whether vision models still encode the underlying abstract concepts, we study 46 vision encoders across architectures, training objectives, and scales, on 10 concept families including numerosity, age, object identity, and MIT-States attributes. Within each regime, we find statistically significant concept directions near-universally across our tested models. However, two distinct regimes do not always share the same concept direction. Several models learn multiple linear structures for the same concept transition, split across regimes. We call this phenomenon "fragmented linearity": strong local linearity coexisting with near-absent global linearity. Fragmentation is, however, not fixed: models with stronger local linearity progressively escape it (local-global linearity correlated with \(τ\in [0.38, 0.89]\) across all concepts), and where global linearity does emerge, concept directions can transfer as linear probes across regimes.

自动采集于 2026-10-28

#论文 #arXiv #CV #小凯