You stand in a room with a table, a chair, and a half-open door. Someone asks: "If you walk to that corner and face the window, will the chair be on your left or right?"
You don't need to actually walk over. You "rotate" in your mind—imagine standing there, facing that direction, and "see" where the chair is.
This is imaginative perception. Humans do it effortlessly, but for current vision-language models (VLMs), it's precisely their fatal weakness.
The Problem: VLMs Can See, But Can't "Imagine"
GPT-5, Gemini 3 Flash, Qwen3-VL—these models are already incredibly strong at image recognition, captioning, and QA. But spatial reasoning trips them up: predicting what a scene looks like from a new angle, what you'd pass along a path, or how many objects appear across multiple views.
The common trait: the answer isn't in the input image. You must "construct" unseen spatial structure from available information.
Existing solutions fall into two camps:
1. Text Chain-of-Thought (CoT): have the model reason through spatial relations step by step in words 2. Visual intermediate representations: depth maps, bounding boxes, etc.
But they share a blind spot—they only process what has already been seen. Depth maps distill the input image; bounding boxes annotate visible objects. True spatial reasoning requires predicting what is not seen.
IPT: Imagine First, Then Answer
A research team from the University of Washington and AI2 proposed Imaginative Perception Tokens (IPT) with a remarkably simple idea:
> Instead of answering spatial questions directly, have the model first "imagine" what would be seen from another spatial configuration, then answer based on that imagination.
IPT splits spatial reasoning into two steps:
1. Imagination stage: given an input image and a spatial query (e.g., "viewed from position X"), the model generates a set of imagination tokens representing the scene from that position 2. Answering stage: the imagination tokens are concatenated with the original input to produce the final answer
Key constraint: imagination isn't free-form—it must stay consistent with the observed scene. You can't imagine a door that doesn't exist, but you can "see" the chair hidden behind the wall.
Three Tasks, Three Kinds of Imagination
The paper designs three tasks, each requiring a different type of imagination:
Perspective Taking (PET): given a first-person photo and a target position, "if you walked there and turned left, would the chair be on your left or right?"—requires imagining the scene from a new position.
Path Tracing (PT): given a top-down map and a path, "what would you see on your left walking along this route?"—requires imagining continuous viewpoints along the path.
Multiview Counting (MVC): given photos from several angles, "how many objects are in the room?"—requires integrating partial observations into a complete mental top-down view.
Each task has ~20K training samples, drawn from real-world scenes and the AI2-THOR simulator, with ground-truth intermediate imagination images and final answers.
Technical Implementation: BAGEL's Dual Life
IPT is built on BAGEL—a unified understanding + generation model. BAGEL uses a Mixture-of-Transformer-Experts (MoT) design: two transformer experts, one for understanding (SigLIP2 ViT encoding) and one for generation (FLUX VAE decoding), cooperating within the same token sequence via shared self-attention.
IPT's cleverness: repurposing BAGEL's image generation capability for spatial reasoning. Generation tokens originally meant for stylized images now generate "scenes from another viewpoint"—not artistic creation, but precise spatial imagination.
Training jointly optimizes two losses:
- Flow-Matching Loss: keeps imagination tokens close to ground-truth target-view images
- Language Modeling Loss: keeps final answers accurate
- Label-only training: 55.8%
- Text CoT training: 53.1% (a 2.7% drop)
- IPT training (answer-only inference): 61.1%
- PT: model imagination 50.4% → ground-truth imagination 86.7% (+36.3%)
- PET: model imagination 96.8% → ground-truth 96.7% (nearly no difference)
The Most Counterintuitive Finding: Text Reasoning Hurts
The most surprising result isn't how well IPT does—it's how badly Text CoT performs.
On Path Tracing (PT):
Describing spatial relations in words is worse than answering directly. The paper's explanation is incisive: spatial computation shouldn't be forced through the language channel. Language is linear and discrete; space is parallel and continuous. Describing "the chair at the door's left-front, 30 degrees up" is verbose and imprecise compared to simply "seeing" the image.
More striking: IPT-trained models, even when *not* generating imagination images at inference (answer-only mode), still beat Text CoT. IPT training builds stronger internal spatial representations—imagination becomes "internal kung fu" rather than "external technique."
Imagination Quality Is the Key Bottleneck
A key ablation: replace model-generated imagination images with ground-truth ones.
Results:
Like a student whose solution approach is fully correct but whose calculations keep going wrong—not a matter of understanding, but of precision.
Head-to-Head with GPT-5
On Path Tracing, IPT-trained BAGEL (answer-only mode) reaches 61.1%, tying GPT-5. On Perspective Taking, IPT hits 96.8%, far surpassing all closed-source models.
On Multiview Counting, IPT improves over baseline by 3.4%—small in absolute terms, but MVC is the hardest of the three tasks (requiring integration of multiple views), so any gain is hard-won.
Transfer experiments are also convincing: on external benchmarks like SAT (perspective taking) and MessyTable (multiview counting), IPT-trained models improved from 34.9% to 63.6% and from 29.0% to 37.0% respectively. The learned spatial reasoning transfers to unseen scenarios.
Engineering Insights
1. Imagine during training; skip it at inference
IPT's biggest engineering value: imagination supervision during training, with optional image generation at inference. Better internal spatial representations with zero inference latency cost—a "train-time investment, inference-time free" paradigm.
2. Unified models beat assembled ones
BAGEL's unified understanding + generation makes IPT natural—no external image generator needed; imagination tokens arise within the same transformer. By contrast, a GPT-4V + DALL-E assembly loses information in cross-model handoffs.
3. Spatial reasoning needs dedicated training data
Generic VLM training data contains almost no "imagine from another angle" tasks. The paper's 3×20K dataset fills this gap—and explains why existing models struggle: not a lack of capability, but a lack of practice.
4. The modality-matching principle
When a task is inherently spatial, intermediate reasoning should be spatial too. Forcing text-based spatial reasoning is like drawing with your mouth—not impossible, just inefficient. This principle may extend widely: symbols for math, images for visual reasoning, spatial representations for spatial reasoning.
Personal Reflection
This paper raises a deeper question: the reasoning bottleneck of current AI may not be reasoning ability itself, but representation choice.
We've been using language as AI's universal reasoning medium—CoT, ToT, GoT are all word games. But human reasoning was never purely linguistic. Mathematicians sketch on paper, architects rotate buildings in their minds, chess players simulate games internally—all non-verbal reasoning.
IPT demonstrates one thing: give AI the right representational medium, and its reasoning capabilities emerge naturally. No longer CoT, no more parameters—just let it "think" in the right modality.
Admittedly, IPT's imagination quality is still the bottleneck (50.4% vs 86.7% on PT), but as generative models improve, that gap will shrink naturally. When AI's imagination becomes precise enough, spatial reasoning will no longer be a problem.
The paper also carries an interesting philosophical implication: imagination is not free fantasy, but constrained reasoning. IPT's imagination must stay consistent with observation—you can't imagine nonexistent objects. This echoes Kant's view of imagination as "the blind but indispensable instrument of the understanding." Imagination isn't anti-logical—it's an extension of logic.
---
Paper: Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Venue: CVPR 2026 MUSI Workshop
Authors: Mahtab Bigverdi, Linjie Li, Weikai Huang, Yiming Liu, Jaemin Cho, Jieyu Zhang, Tuhin Kundu, Chris Dongjoo Kim, Zelun Luo, Ranjay Krishna, Linda Shapiro
Code: Not yet released