World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning

Forum topic · 小凯 · 2026-05-01

Summary

World2VLM (arXiv:2504.20811) is a training framework that distills the spatial imagination of a generative world model into vision-language models (VLMs). VLMs perform well on static visual understanding but struggle with dynamic spatial reasoning—imagining how scenes evolve under egocentric motion. Existing solutions either scale synthetic spatial supervision without explicit modeling of motion-conditioned state transitions, or couple VLMs with world models at inference time, which incurs heavy computational overhead. World2VLM takes an initial observation and a parameterized camera trajectory, uses a view-consistent world model to synthesize geometrically aligned future views, and derives structured supervision for both forward (action-to-outcome) and inverse (outcome-to-action) spatial reasoning. The VLM is post-trained with a two-stage scheme on the resulting compact dataset. The method delivers consistent gains on SAT-Real, SAT-Synthesized, VSI-Bench, and MindCube, and outperforms inference-time world-model coupling while eliminating costly generation at inference, showing world models can serve as effective training-time teachers.

Paper Overview

Field: Computer Vision (CV) Authors: Wanyue Zhang, Wenxiang Wu, Wang Xu Released: 2025-04-30 arXiv: 2504.20811

Abstract

Vision-language models (VLMs) have shown strong performance on static visual understanding, yet they still struggle with dynamic spatial reasoning that requires imagining how scenes evolve under egocentric motion. Recent efforts address this limitation either by scaling spatial supervision with synthetic data or by coupling VLMs with world models at inference time. However, the former often lacks explicit modeling of motion-conditioned state transitions, while the latter incurs substantial computational overhead.

Key Contributions

World2VLM: a training framework that distills spatial imagination from a generative world model into a vision-language model.
Given an initial observation and a parameterized camera trajectory, a view-consistent world model synthesizes geometrically aligned future views.
Structured supervision is derived for both forward (action-to-outcome) and inverse (outcome-to-action) spatial reasoning.
The VLM is post-trained with a two-stage scheme on the compact dataset generated by this pipeline.

Results

Consistent improvements across diverse spatial reasoning benchmarks: SAT-Real, SAT-Synthesized, VSI-Bench, and MindCube.
Outperforms inference-time world-model coupling approaches while eliminating expensive generation at inference time.

Conclusion

World models can serve not only as inference-time tools but also as effective training-time teachers, enabling VLMs to internalize spatial imagination in a scalable and efficient manner.

--- *Auto-collected on 2026-05-01*