Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Models (arXiv 2606.03988)

Forum topic · 小凯 · 2026-06-04

Summary

Vision-language models (VLMs) perform well on many tasks but struggle with spatial reasoning when key information is not directly observable. A new paper (arXiv:2606.03988) introduces Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while staying consistent with observed input. The authors formalize three benchmark tasks—Perspective Taking (PET), Path Tracing (PT), and Multiview Counting (MVC)—and build datasets of roughly 20K examples containing ground-truth imaginative content, answers, and evaluation benchmarks. Using the unified VLM BAGEL as a backbone, IPT supervision consistently improves spatial reasoning and often outperforms text-based chain-of-thought training. Notably, the gains hold even when no images need to be generated at inference time. Authored by researchers including Mahtab Bigverdi, Lindsey Li, and Ranjay Krishna, the work suggests that teaching models to represent hypothetical views is a promising path toward stronger spatial intelligence in multimodal AI systems.

Paper Overview

Field: Machine Learning
Authors: Mahtab Bigverdi, Lindsey Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dangjoo Kim, Zelun Luo, Linda Shapiro, Ranjay Krishna
arXiv: 2606.03988

Abstract

Vision language models (VLMs) excel at many tasks but still struggle with spatial reasoning when critical information is not directly observable. Many such problems require imaginative perception: inferring what would be seen from an unseen viewpoint, tracing paths through occluded spaces, or integrating partial observations into a coherent spatial representation.

The authors introduce Imaginative Perception Tokens (IPT) — intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations while remaining consistent with the observed input.

Tasks and Datasets

To study this capability, the paper formulates three tasks:

1. Perspective Taking (PET) — inferring what would be seen from an unseen viewpoint 2. Path Tracing (PT) — tracing paths through occluded spaces 3. Multiview Counting (MVC) — integrating partial observations into a coherent spatial representation

Datasets of approximately 20K examples are constructed, including ground-truth imaginative content, answers, and evaluation benchmarks.

Results

Using the unified VLM BAGEL as a backbone, IPT supervision consistently improves spatial reasoning and often outperforms text-based chain-of-thought training — even when no images need to be generated at inference time.

--- *Auto-collected on 2026-06-04. Source: arXiv:2606.03988*