Astra: Agentic Visual Spatial Reasoning with World Simulators

Forum topic · 小凯 · 2026-06-07

Summary

Astra is an agentic spatial reasoning framework that enables vision-language models (VLMs) to reason through imagination by actively acquiring imagined visual evidence from a world simulator during inference. The system couples Astra-VL, an RL-trained VLM policy, with Astra-WM, a Bagel-based world simulator that generates novel-view observations from context images and natural-language camera motions, refined via view-consistency tuning for cross-view pose and content alignment. A two-stage, world-simulator-in-the-loop RL curriculum stabilizes tool-use exploration and teaches the model to invoke the simulator only when imagined observations improve over direct answers. Experiments show both components matter: Astra-WM boosts simulator-augmented Gemini-3-Flash on MMSI-Bench from 45.1 to 49.5, while Astra-VL improves the Qwen3-VL backbone from 29.8 to 38.8 on MMSI-Bench and from 36.8 to 42.7 on MindCube. Results indicate imagined observations provide useful spatial evidence, but effective world-model-augmented reasoning requires learning when, where, and how to imagine. arXiv: 2606.06476

Overview

Field: Computer Vision (CV)
Authors: Chenming Zhu, Jingli Lin, Yilin Long
Published: 2026-06-04
arXiv: 2606.06476

Problem

While Vision-Language Models (VLMs) have shown strong visual reasoning capabilities, their spatial reasoning remains largely constrained to observed images and text-oriented chain-of-thought. They struggle to:

Infer unobserved layouts
Maintain cross-view consistency
Reason from alternative viewpoints with only limited egocentric observations

Approach: Thinking with Imagination

The authors frame this problem as *thinking with imagination*, where a VLM actively acquires imagined visual evidence by interacting with a world simulator during reasoning. Astra is an agentic spatial reasoning framework giving VLMs action-conditioned visual imagination, coupling:

Astra-VL: an RL-trained VLM policy
Astra-WM: a Bagel-based world simulator that generates novel-view observations from context images and natural-language camera motion commands. It is trained with view-consistency tuning to improve cross-view pose and content consistency of imagined evidence.

For training, the paper proposes a two-stage, world-simulator-in-the-loop RL curriculum that stabilizes tool-use exploration and teaches the model to invoke the simulator only when imagined observations are superior to answering directly.

Results

| Setting | Benchmark | Before | After | |---|---|---|---| | Astra-WM + Gemini-3-Flash | MMSI-Bench | 45.1 | 49.5 | | Astra-VL (Qwen3-VL backbone) | MMSI-Bench | 29.8 | 38.8 | | Astra-VL (Qwen3-VL backbone) | MindCube | 36.8 | 42.7 |

Takeaway

Both the world simulator and the agentic policy are necessary: imagined observations can supply useful spatial evidence, but effective world-model-augmented reasoning requires learning when, where, and how to imagine.

*Source: arXiv:2606.06476*