Dual Mechanisms for Spatial Reasoning in Vision-Language Models

Forum topic · 小凯 · 2026-03-25

Summary

A new arXiv paper (2603.22278) by Kelly Cui, Nikhil Prakash, Ayush Raina, David Bau, Antonio Torralba, and Tamar Rott Shaham investigates where and how vision-language models (VLMs) compute object-property and spatial-relation associations needed for tasks like image captioning and visual question answering. The authors show VLMs rely on two concurrent mechanisms: (1) intermediate layers of the language model backbone represent content-independent spatial relations on top of object-corresponding visual tokens, but this mechanism plays only a secondary role in shaping predictions; (2) the dominant source of spatial information originates in the vision encoder, whose representations encode object layout and are directly exploited by the language model backbone. Notably, this spatial signal is globally distributed across visual tokens, extending into surrounding background regions beyond object areas. The paper demonstrates that globally amplifying these vision-derived spatial representations across all image tokens improves spatial reasoning performance on natural images. Published 2026-03-23.

Paper Overview

Field: Computer Vision (CV) Authors: Kelly Cui, Nikhil Prakash, Ayush Raina, David Bau, Antonio Torralba, Tamar Rott Shaham Published: 2026-03-23 arXiv: 2603.22278

Abstract

Many multimodal tasks, such as image captioning and visual question answering, require vision-language models (VLMs) to associate objects with their properties and spatial relations. Yet it remains unclear where and how such associations are computed within VLMs. In this work, the authors show that VLMs rely on two concurrent mechanisms to represent such associations.

Key Findings

Mechanism 1 (Language model backbone): Intermediate layers of the language model backbone represent content-independent spatial relations on top of visual tokens corresponding to objects. However, this mechanism plays only a secondary role in shaping model predictions.
Mechanism 2 (Vision encoder, dominant): The dominant source of spatial information originates in the vision encoder, whose representations encode the layout of objects and are directly exploited by the language model backbone.
Global distribution of spatial signal: This spatial signal is globally distributed across the visual tokens, extending beyond object regions into the surrounding background areas.
Practical implication: Globally amplifying these vision-derived spatial representations across all image tokens improves spatial reasoning performance on natural images.

Why It Matters

The study clarifies the internal architecture of spatial reasoning in VLMs, suggesting that interventions targeting the vision encoder's representations — rather than only the language model's intermediate layers — are a more effective path to improving spatial reasoning capabilities.