When humans are asked "if you walk to that corner and face the window, will the chair be on your left or right?", we don't physically move—we mentally rotate. This is imaginative perception, and it is exactly where current vision-language models (VLMs) like GPT-5, Gemini 3 Flash, and Qwen3-VL fall short: they can describe what they see, but they cannot *predict* what they would see from another viewpoint, along a path, or across multiple views.
The Problem: Existing Methods Only Process What Is Visible
Common workarounds fall into two camps, both with the same blind spot:
1. Text chain-of-thought (CoT): reasoning about spatial relations step by step in words 2. Visual intermediate representations: generating depth maps or bounding boxes
Both only refine *already-seen* information. True spatial reasoning requires predicting what is not seen.
IPT: Imagine First, Answer Second
A team from the University of Washington and AI2 proposes Imaginative Perception Tokens (IPT):
1. Imagination stage: given an input image and a spatial query (e.g., "view from position X"), the model generates imagination tokens representing what would be seen from that configuration 2. Answer stage: the imagination tokens are concatenated with the original input to produce the final answer
Crucially, the imagination is *constrained*: it must stay consistent with the observed scene—you cannot imagine a door that does not exist, but you can "see" the chair hidden behind a wall.
Three Tasks, Three Kinds of Imagination
- Perspective Taking (PET): from a first-person photo plus a target location, answer whether an object would be on the left or right after moving and turning
- Path Tracing (PT): given a top-down map and a path, describe what appears on the left while walking it
- Multiview Counting (MVC): integrate several photos from different angles to count total objects in a room
- Flow-Matching Loss: pushes imagination tokens toward ground-truth target-view images
- Language Modeling Loss: optimizes final answer accuracy
- Label-only training: 55.8%
- Text CoT training: 53.1% (a 2.7% drop)
- IPT training (answer-only inference): 61.1%
- PT: 50.4% → 86.7% (+36.3%)
- PET: 96.8% → 96.7% (essentially unchanged)
- PT: IPT-trained BAGEL (answer-only) reaches 61.1%, matching GPT-5
- PET: IPT reaches 96.8%, far exceeding all closed-source models
- MVC: +3.4% over baseline on the hardest task
- Transfer: SAT 34.9% → 63.6%; MessyTable 29.0% → 37.0%, showing learned spatial reasoning generalizes
Each task has ~20K training samples drawn from real-world data and the AI2-THOR simulator, with ground-truth intermediate imagination images and final answers.
Technical Implementation: BAGEL's Dual Role
IPT is built on BAGEL, a unified understanding + generation model using a Mixture-of-Transformer-Experts (MoT) design: one expert handles understanding (SigLIP2 ViT encoder), the other generation (FLUX VAE decoder), cooperating via shared self-attention over a single token sequence. IPT repurposes BAGEL's image generation tokens for precise spatial imagination rather than artistic creation.
Training uses two joint losses:
The Counterintuitive Finding: Text CoT Hurts
On Path Tracing:
Verbalizing spatial relations is worse than answering directly—language is linear and discrete, while space is parallel and continuous. Even more striking: an IPT-trained model that *skips* generating imagination images at inference still beats Text CoT, meaning IPT training builds stronger internal spatial representations—imagination becomes internal kung fu rather than an external crutch.
Imagination Quality Is the Bottleneck
Replacing model-generated imagination images with ground-truth ones:
On PT, the model "knows" how to reason but "imagines" imprecisely—like a student with the right approach but sloppy arithmetic.
Comparison with GPT-5 and Transfer
Engineering Insights
1. Imagine during training, optionally skip at inference — better internal representations with no inference-time latency cost: a "train-time investment, inference-free" paradigm. 2. Unified models beat assembled pipelines — no need to bolt on an image generator; cross-model information transfer losses are avoided (unlike GPT-4V + DALL-E style setups). 3. Spatial reasoning needs dedicated training data — generic VLM data contains almost no "imagine from another viewpoint" tasks; the 3×20K dataset fills this gap. 4. Modality matching principle — when a task is fundamentally spatial, intermediate reasoning should be spatial too. This may generalize: symbolic for math, images for visual reasoning, spatial representations for spatial reasoning.
Takeaway
The deeper lesson: AI's reasoning bottleneck may not be reasoning ability itself, but representation choice. Give AI the right representational medium and reasoning emerges naturally—no longer CoT, no extra parameters, just thinking in the correct modality. Current imagination quality remains the bottleneck (50.4% vs 86.7% on PT), but as generative models improve, that gap should shrink. Philosophically, IPT frames imagination not as free fantasy but as *constrained reasoning*—it must remain consistent with observation, echoing Kant's view of imagination as an indispensable instrument of the understanding.
---
Paper: Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Venue: CVPR 2026 MUSI Workshop
Authors: Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dongjoo Kim, Zelun Luo, Ranjay Krishna, Linda Shapiro
Code: not yet released