VLM-IE3D: 3D-Aware Vision-Language Models with Implicit and Explicit Geometries

Forum topic · 小凯 · 2026-07-27

Summary

Most existing vision-language models (VLMs) built on 2D visual inputs struggle with 3D tasks that require fine-grained spatial understanding and reasoning. VLM-IE3D is a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned purely from RGB videos. The method introduces Implicit Geometry Tokens (IGTs), which capture high-level geometric priors from input videos, and complementary Explicit Geometry Tokens (EGTs), which encode detailed geometric structures extracted from reconstructed 3D attributes. A 3D-aware adapter effectively fuses these two geometric representations with 2D visual cues, injecting strong 3D inductive biases without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior and consistent performance across diverse 3D tasks, including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Paper by Wenhao Li, Xueying Jiang, and Quanhao Qian, arXiv:2507.21747.

Paper Overview

Field: Computer Vision Authors: Wenhao Li, Xueying Jiang, Quanhao Qian Published: 2025-07-27 arXiv: 2507.21747

Abstract

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, the authors present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos.

Key Contributions

Implicit Geometry Tokens (IGTs): capture high-level geometric priors from input videos.
Explicit Geometry Tokens (EGTs): encode detailed geometric structures from reconstructed 3D attributes, complementing IGTs.
3D-aware adapter: effectively fuses the two types of geometric representations with 2D visual cues.

Highlights

This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning, without requiring any additional 3D inputs.
Extensive experiments demonstrate superior and consistent performance across a range of 3D tasks:
3D video detection
3D visual grounding
3D dense captioning
Spatial reasoning