VEGA-3D: Unlocking Implicit 3D Priors in Video Generation Models for Spatial Understanding

Forum topic · 小凯 · 2026-03-22

Summary

Researchers from Huazhong University of Science and Technology and Baidu introduced VEGA-3D, a framework that addresses 'spatial blindness' in multimodal large language models (MLLMs). While MLLMs excel at semantic understanding, they struggle with fine-grained geometric reasoning due to contrastive pretraining and the scarcity of annotated 3D data. The key insight: video generation models must internally learn 3D structure and physical dynamics to produce temporally coherent video, making them implicit 'latent world simulators'. VEGA-3D repositions a pretrained video diffusion model (e.g., Wan2.1) as a second visual encoder, extracting spatiotemporal features at intermediate noise levels via DDIM inversion, then fusing them with SigLIP semantic features through token-level adaptive gated fusion. Without any 3D-annotated training data, VEGA-3D improves over the Video-3D LLM baseline across six benchmarks: ScanRefer Acc@0.5 rises from 51.7% to 56.2%, SQA3D EM from 58.6% to 61.3%, and VSI-Bench relative direction/distance accuracy by over 13 points. The work suggests a paradigm shift from explicit 3D annotation to distilling implicit generative knowledge, with implications for embodied AI and unified generation-understanding models.

Overview

Multimodal large language models (MLLMs) like GPT-4V, Claude 3, and Gemini can describe images well but suffer from "spatial blindness"—they recognize objects yet fail at fine-grained geometric reasoning (relative positions, distances, camera viewpoints). The root cause: MLLM vision comes from contrastive learning (e.g., SigLIP), which teaches semantics but not geometry, while annotated 3D data (point clouds, depth maps, poses) is expensive and scarce.

Key idea

Video generation models must already possess implicit 3D knowledge: to generate coherent video, they must maintain a stable 3D world model—handling object permanence, parallax, occlusion, and perspective. The authors verified this via multi-view consistency experiments on video diffusion models (Wan2.1, Vmem), finding strong feature correspondence, structured PCA patterns across viewpoints, and correct occlusion handling.

VEGA-3D architecture

A dual-branch visual encoding system:

Semantic branch: SigLIP extracts semantic features (what objects are, what they're doing).
Generative branch ("Latent World Simulator"): A frozen pretrained video diffusion model re-purposed as a feature extractor. Using DDIM inversion, real images are noised to an intermediate diffusion level (empirically t≈300–500 of 1000), where features retain structural information without being fully "solidified." This level proved richest in 3D priors.
Token-level adaptive gated fusion: For each visual token, learned gates dynamically weight semantic vs. 3D features (fusion = gate_sem × sem + gate_3d × 3D), resolving distribution shift (via LayerNorm + projection layers) and information conflicts.

Results

Across benchmarks (ScanRefer, Scan2Cap, SQA3D, VSI-Bench, embodied manipulation), without any 3D-annotated data:

ScanRefer: Acc@0.25 68.5 → 71.2; Acc@0.5 51.7 → 56.2 (+4.5)
SQA3D: EM 58.6 → 61.3 (+2.7)
VSI-Bench: appearance order 52.1% → 61.8%; relative direction 45.3% → 58.7% (+13.4); relative distance 48.9% → 62.4% (+13.5)

Ablations confirm: removing the 3D branch reverts to baseline; intermediate noise levels outperform early/late extraction; adaptive gating beats simple concatenation by 2–3 points; results are robust across different underlying video generation models. Qualitatively, attention maps show VEGA-3D's attention is sharply focused on target objects, while the baseline is diffuse. Failure cases involve distinguishing nearly identical instances.

Significance

Paradigm shift: from explicit 3D annotation to distilling implicit knowledge already learned by generative models—bypassing the 3D data bottleneck.
Scalability: the framework improves automatically as video generation models improve; future work may extract dynamic physics, causal, and cross-modal priors.
Embodied AI: the learned priors encode physical intuition useful for manipulation and navigation.
Theory: supports the view that generative ability may be a superset of understanding ability, pointing toward unified generative-discriminative models.

References

Paper: Wu, X., Liang, D., Feng, T., et al. "Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding." arXiv:2603.19235
Code: https://github.com/H-EmbodVis/VEGA-3D