Do Vision Models Know That Three Is Three? Fragmented Linearity in Visual Concept Representations

Forum topic · 小凯 · 2026-10-27

Summary

This paper by Ankit Sonthalia, Arnas Uselis, and Seong Joon Oh (arXiv:2610.35857) investigates whether vision models encode abstract concepts in a shared linear direction across visually unrelated contexts, as the linear representation hypothesis (LRH) suggests for language models. The authors evaluate 46 vision encoders across architectures, training objectives, and scales on 10 concept families, including numerosity, age, object identity, and MIT-States attributes. Within a single regime (e.g., handwritten digits), statistically significant concept directions exist almost universally. However, when the same concept transition appears in different regimes (e.g., the 2→3 change in digits versus hand gestures), models do not always share one direction: several learn multiple linear structures split across regimes, a phenomenon the authors call 'fragmented linearity'—strong local linearity with near-absent global linearity. Fragmentation is not permanent: models with stronger local linearity progressively escape it, with local-global linearity correlated at τ ∈ [0.38, 0.89]. Where global linearity emerges, concept directions transfer as linear probes across regimes.

Paper Overview

Field: Computer Vision Authors: Ankit Sonthalia, Arnas Uselis, Seong Joon Oh Published: 2026-10-26 arXiv: 2610.35857

Abstract

Do vision models encode the transition $2{\to}3$ the same way whether it appears in handwritten digits or in hand gestures, and do they "understand" that clean ${\to}$ dirty is the same transformation regardless of whether it is applied to a kitchen or a carpet? The linear representation hypothesis (LRH) answers this for language models: concept changes correspond to directions in (language) embedding space. In vision, the same concept can manifest in visually unrelated "regimes" that rarely co-occur: images of digits and hands do not share much in common in pixel space.

To understand whether vision models still encode the underlying abstract concepts, we study 46 vision encoders across architectures, training objectives, and scales, on 10 concept families including numerosity, age, object identity, and MIT-States attributes. Within each regime, we find statistically significant concept directions near-universally across our tested models. However, two distinct regimes do not always share the same concept direction. Several models learn multiple linear structures for the same concept transition, split across regimes. We call this phenomenon "fragmented linearity": strong local linearity coexisting with near-absent global linearity.

Fragmentation is, however, not fixed: models with stronger local linearity progressively escape it (local-global linearity correlated with $\tau\in [0.38, 0.89]$ across all concepts), and where global linearity does emerge, concept directions can transfer as linear probes across regimes.

Key Takeaways

Local linearity is nearly universal: Within a regime, almost all tested vision encoders exhibit statistically significant concept directions.
Global linearity is often missing: The same concept change (e.g., 2→3) learned in one regime does not always align with the direction in another regime.
Fragmented linearity: Multiple linear structures for the same concept transition can coexist across regimes in a single model.
Fragmentation scales with model quality: Stronger local linearity correlates with less fragmentation ( $\tau \in [0.38, 0.89]$ ).
Transfer where it matters: When global linearity emerges, concept directions work as linear probes that transfer across regimes.

---

*Automatically collected on 2026-10-28.*