WanderDream: Letting AI Wear the Shoes of Thought — The First Large-Scale Dataset for Emulative Imagination

> Robots can't climb stairs; visually impaired people hesitate to move around blindly — if AI can't physically go somewhere, can it "imagine" what lies ahead?

• Paper: https://arxiv.org/abs/2603.06445
• Authors: Ruiping Liu et al. (KIT, Hunan University, ETH Zürich, etc.)
• Published: arXiv, March 2026

The Core Problem: Knowing Without Going

Embodied AI assumes that understanding an environment requires active exploration. But this often fails in reality:

• Robots face physical limits — wheeled platforms can't climb stairs or fit through narrow gaps
• Visually impaired individuals have safety concerns — without haptic feedback, they dare not enter unknown areas
• Remote users want to know what a room looks like without being there

The traditional fix is "map + sensors," but mapping itself requires exploration. WanderDream asks: given only a limited initial observation, can an agent mentally simulate the full visual trajectory toward a target scene and answer "what would I see if I walked there?" spatial reasoning questions?

Two Layers of Mental Simulation

Drawing on cognitive science, the paper distinguishes:

| Layer | Name | Purpose | Prior Work | |-------|------|---------|-----------| | 1 | Instrumental Simulation | Task-oriented, aids decision-making | PathDreamer, WMNav, DreamWalker | | 2 | Emulative Simulation | Experience-oriented, answers "what-if" | Nearly blank — WanderDream is first |

• Instrumental: "To reach the chair, turn left and move forward"
• Emulative: "If I walk over, what will I see? Is the sofa on the left or right?"

The first is navigation planning; the second is situated experience. WanderDream targets the latter.

Dataset Structure: Two Complementary Tracks

WanderDream-Gen (Generation)

• 15.8K panoramic videos covering 1,088 real scenes (HM3D, ScanNet++, real-world captures)
• Each video depicts an imagined trajectory from a current viewpoint to a target situation
• Includes depth maps, semantic segmentation, camera poses
• Paths planned via a 3D Probabilistic Roadmap (PRM) — navigating around impassable areas like stairs — rendered from six directions and stitched into 360° panoramic video (21 frames per video)

WanderDream-QA (Reasoning)

• 158K QA pairs across 10 question types at three levels: start state (s₀), path (s₀→sₜ), and goal state (sₜ)
• Generated with GPT-5 grounded in ground-truth scene annotations, covering both robot and human perspectives

Four Key Experimental Findings

1. Mental exploration is essential for situated reasoning — without imagination, models perform poorly on spatial reasoning from current observation alone. 2. World models perform impressively on WanderDream-Gen, generating coherent imagined trajectories. 3. Imagination boosts reasoning — using generated imagination videos to assist QA substantially outperforms answering from static images. 4. Strong sim-to-real transfer — on a real-world test set (26 videos, 182 QA pairs) recorded with panoramic head-mounted cameras, models trained on simulated data still perform well under occlusion and imperfect trajectories.

Technical Details at a Glance

• Robot contexts: 19 object categories in HM3D as navigation anchors; start points sampled 1.5–3 m from targets
• Human contexts: three states in ScanNet++ (standing, sitting, interacting) with random orientation and distance
• Video: six-direction views stitched with the PanoContext toolbox; adaptive frame rate based on path length
• QA: GPT-5 with carefully designed prompts, grounded in ground truth

Why It Matters

• Fills the data gap for experience-oriented imagination, beyond navigation planning and static scene understanding
• Practical value for vulnerable groups: imagine AI "pre-walking" a route for a visually impaired user — contextualized safety information, not just navigation commands
• Enables zero-exploration reasoning: spatial reasoning without physically visiting the space

Limitations

• Limited scale: 1,088 scenes vs. the full HM3D corpus
• Inconsistent generation quality: temporal flickering in generated videos
• Small real-world test set: 26 videos only validate sim-to-real preliminarily
• Question types skew spatial; causal reasoning ("what if I knock over this vase?") is not covered
• Potential biases from GPT-5-generated QA

Related Work Comparison

| Work | Type | Difference from WanderDream | |------|------|------------------------------| | PathDreamer | Instrumental (navigation) | Predicts future frames, no what-if QA | | MindJourney | Step-wise visual imagination | Viewpoint changes, no goal-directed trajectories | | GenEx | Forward panoramic video | Current-state reasoning only | | SQA3D / MSQA | Static scene QA | Requires pre-explored static scenes | | Situat3DChange | Temporal change | Scene changes, not path imagination |

One-Sentence Takeaway

> WanderDream is the first large-scale dataset built for emulative imagination, enabling AI to mentally simulate complete visual paths and answer spatial reasoning questions without physically exploring — opening up "zero-exploration reasoning" for constrained robots and assistive technologies.

References

• Paper: https://arxiv.org/abs/2603.06445
• Project page/data: to be released (per the paper)
• HM3D: https://aihabitat.org/datasets/hm3d/
• ScanNet++: http://www.scan-net.org/