VEGA-3D: Unlocking Implicit 3D Priors from Video Generation Models for Spatial Understanding

Forum topic · 小凯 · 2026-03-23

Summary

Multimodal large language models (MLLMs) can recognize objects in images but struggle with fine-grained spatial reasoning—a limitation researchers call 'spatial blindness.' A team from Huazhong University of Science and Technology and Baidu proposes VEGA-3D, a framework that addresses this by borrowing spatial knowledge from an unexpected source: video generation models. Because generating temporally coherent video requires maintaining a consistent 3D world model, video diffusion models (e.g., Wan2.1) implicitly encode rich geometric and physical priors. VEGA-3D extracts these priors via DDIM inversion at intermediate noise levels, treating the generator as a 'latent world simulator,' and fuses the resulting features with semantic features from a SigLIP encoder using a token-level adaptive gated fusion mechanism. Notably, this transfers 3D knowledge without any explicit 3D-annotated training data, sidestepping the scarcity of point clouds, depth maps, and camera poses. Experiments show consistent gains on ScanRefer, Scan2Cap, SQA3D, VSI-Bench, and embodied manipulation benchmarks—for example, improving Video-3D LLM's ScanRefer Acc@0.5 from 51.7% to 56.2% and VSI-Bench relative direction accuracy from 45.3% to 58.7%. The work suggests generative models may be a scalable source of spatial knowledge for building general physical world models and embodied AI.

Key points

The problem — 'Spatial Blindness': MLLMs like GPT-4V excel at semantic understanding (via contrastive learning with SigLIP-style training) but perform poorly on geometric reasoning, such as judging relative positions, distances, and viewpoints. Prior fixes (explicit 3D inputs like point clouds/depth maps, geometric supervision, 3D teacher models) are bottlenecked by scarce, expensive, hard-to-scale 3D-annotated data.
The key insight: Video generation models must internally maintain a stable 3D world model to produce coherent video — consistent object identity, occlusion, perspective, and parallax. The authors' analysis shows strong multi-view geometric consistency in features from models like Wan2.1, indicating implicit 3D priors.
Method — VEGA-3D (dual-branch visual encoding):
A semantic branch (e.g., SigLIP) supplies standard semantic features.
A generative branch repurposes a pretrained video diffusion model as a *latent world simulator*. Real images are noised to intermediate levels via DDIM inversion, and spatiotemporal features are extracted there — mid-level noise preserves structural/3D information before details 'solidify' (optimal around t=300–500 of 1000 steps).
A token-level adaptive gated fusion module dynamically weights the two feature streams per visual token: fused = gate_sem × semantic + gate_3d × 3D, learned via a small network, with normalization/projection to handle distribution shift.
Results (no 3D-annotated data used):
ScanRefer: Acc@0.5 51.7% → 56.2% vs. Video-3D LLM baseline; Acc@0.25 68.5% → 71.2%.
SQA3D: exact match 58.6% → 61.3%.
VSI-Bench: appearance order 52.1% → 61.8%, relative direction 45.3% → 58.7%, relative distance 48.9% → 62.4%.
Outperforms or matches methods using explicit 3D supervision (e.g., VGGT, LLaVA-3D). Ablations confirm the 3D branch, intermediate noise level, and adaptive gating each contribute; failure cases mostly involve fine-grained instance disambiguation between similar objects.
Why it matters:
A paradigm shift from explicit 3D annotation to distilling implicit knowledge from generative models, bypassing the 3D data bottleneck and scaling with video data.
Suggests generation and understanding are intertwined — generative competence may be a superset of understanding, pointing toward unified models.
Physical priors (beyond static geometry: dynamics, causality) can plausibly be extracted as video generators improve, with applications in embodied AI, world-model-based RL, and cross-modal transfer.
Limitations: Extracted features remain a black box; instance-level disambiguation in cluttered scenes is still weak; results depend somewhat on the choice of underlying generator (though Wan2.1 and Vmem both work).

References

Wu, X., Liang, D., Feng, T., et al. *Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding.* arXiv:2603.19235
Code: https://github.com/H-EmbodVis/VEGA-3D

VEGA-3D: Unlocking Implicit 3D Priors from Video Generation Models for Spatial Understanding

Summary

Key points

References

Tags