Overview
WanderDream addresses a core question in embodied AI: can an agent reason about a scene it cannot physically explore? Robots face physical limits (wheels can't climb stairs), and visually impaired users face safety concerns when moving into unknown areas. Traditional approaches assume active exploration to build maps — but what if exploration itself is impossible?
WanderDream asks: given limited initial observation, can an agent mentally simulate the full visual trajectory to a goal scenario and answer spatial "what-if" questions?
- Paper: https://arxiv.org/abs/2603.06445
- Authors: Ruiping Liu et al. (KIT, Hunan University, ETH Zurich, etc.)
- Published: arXiv, March 2026
- 15.8K panoramic videos covering 1,088 real scenes from HM3D (robot navigation), ScanNet++ (human activity scenes), and real-world captures
- Each video depicts an imagined trajectory from the current viewpoint to a target scenario
- Includes depth maps, semantic segmentation, and camera poses
- Paths are planned with a 3D Probabilistic Roadmap (PRM) for shortest routes; impassable areas (e.g., stairs) are automatically bypassed; six-direction views are stitched into 360° panoramas (21 frames per video)
- 158K question-answer pairs across 10 QA types at three levels: start state (s₀), traversed path (s₀→sₜ), and goal state (sₜ)
- QA generated by GPT-5 grounded in ground-truth annotations, covering both robot and human perspectives
- Robot scenarios: 19 target object categories in HM3D as navigation anchors; start points sampled 1.5–3m from targets
- Human scenarios: three states in ScanNet++ (standing, sitting, interacting) with random orientations and distances
- Video generation: six-direction views (front/right/back/left/up/down) stitched via the PanoContext toolbox; frame rate adapts to path length
- QA generation: GPT-5 with carefully designed prompts, distinguishing robot-perspective and human-perspective questions
- Fills the data gap for experience-oriented imagination, combining dynamic imagination with path reasoning at scale
- Practical benefit for vulnerable groups: for visually impaired users, an AI that can "rehearse" a path and report contextual safety information (e.g., "stairs 3 meters ahead, turn right") is more valuable than plain navigation instructions
- Opens zero-exploration reasoning: meaningful spatial inference is possible even without physical exploration
- Scene coverage (1,088) is still limited relative to full HM3D
- Generated videos sometimes show temporal inconsistency and object flicker
- The real-world test set (26 videos) only supports preliminary sim-to-real validation
- QA types focus on spatial reasoning; complex causal reasoning (e.g., "what if I knock over the vase?") is not covered
- GPT-5-generated QA may contain templated phrasing biases
- Paper: https://arxiv.org/abs/2603.06445
- Project page / data: to be released (open-sourcing mentioned in the paper)
- HM3D: https://aihabitat.org/datasets/hm3d/
- ScanNet++: http://www.scan-net.org/
Key Concept: Two Levels of Mental Simulation
Drawing on cognitive science, the paper distinguishes:
| Level | Name | Purpose | Prior work | |---|---|---|---| | 1 | Instrumental Simulation | Task-oriented, aids decision-making | PathDreamer, WMNav, DreamWalker | | 2 | Emulative Simulation | Experience-oriented, answers what-if questions | Nearly blank — WanderDream is the first |
Instrumental simulation means "to reach the chair, turn left then walk forward." Emulative simulation means "if I walked there, what would I see — is the sofa on the left or right?" WanderDream focuses on the latter.
Dataset Structure: Dual-Track Benchmark
WanderDream-Gen (Generation)
WanderDream-QA (Reasoning)
Four Key Findings
1. Mental exploration is essential for situated reasoning — without imagination, models relying only on current observations perform poorly on spatial reasoning. 2. World models perform impressively on WanderDream-Gen, generating fairly coherent imagined trajectories. 3. Imagination significantly boosts reasoning — using generated imagined videos to assist QA substantially outperforms answering from static images. 4. Strong sim-to-real transfer — a real-world test set (26 videos, 182 QA pairs) recorded with a panoramic head-mounted camera shows models trained on synthetic data remain robust under real occlusions and imperfect trajectories.
Technical Details
Why It Matters
Comparison with Related Work
| Work | Type | Difference from WanderDream | |---|---|---| | PathDreamer | Instrumental (navigation) | Predicts future frames only, no what-if QA | | MindJourney | Step-wise visual imagination | Viewpoint changes only, not goal-directed trajectories | | GenEx | Forward panoramic video | Reasons about current state only, no goal scenario | | SQA3D / MSQA | Static scene QA | Requires pre-explored static scenes, no imagination | | Situat3DChange | Temporal change | Scene changes, not path imagination |
Limitations
One-Sentence Summary
WanderDream is the first large-scale dataset built for emulative imagination, enabling AI to mentally simulate complete visual paths and answer spatial reasoning questions without physically traveling — opening "zero-exploration reasoning" possibilities for constrained robots and visually impaired assistance.