Probing Spatial State Representations in Visual Foundation Models

Forum topic · 小凯 · 2026-10-30

Summary

This paper investigates whether visual foundation models maintain a persistent spatial state—an internal representation of the surrounding world that is updated under self-motion and used to guide action—rather than merely recognizing individual images. The authors evaluate four levels of spatial capability: spatial structure, object persistence, predictive spatial update, and actionability. Using frozen-feature probes, they test ten visual representations under streaming egocentric observations of static indoor environments. Results show that capability profiles vary across tasks, network layers, and prefix lengths, and that strong readout of the visible scene does not guarantee accurate object persistence or predictive spatial updating beyond the current view. The paper was posted to arXiv (2610.40636) by Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, and Ziwei Liu.

Overview

Field: Computer Vision
Authors: Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang, Ziwei Liu
Posted: 2026-10-29
arXiv: 2610.40636

Abstract (English, translated)

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state?

The authors study this question through four levels of spatial capability—spatial structure, object persistence, predictive spatial update, and actionability—using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments.

Key Findings

Capability profiles vary across tasks, network layers, and prefix lengths.
Strong readout of the currently visible scene does not ensure accurate object persistence or predictive updating of unseen parts of the environment.