VLM-IE3D: 3D-Aware Vision-Language Models with Implicit and Explicit Geometries

Forum topic · 小凯 · 2026-07-25

Summary

This paper introduces VLM-IE3D, a unified framework that enhances the 3D spatial awareness of vision-language models (VLMs) using only RGB video input. Existing VLMs built on 2D visual inputs often struggle with 3D tasks requiring fine-grained spatial understanding and reasoning. VLM-IE3D addresses this by introducing Implicit Geometry Tokens (IGTs), which capture high-level geometric priors from input videos, and complementary Explicit Geometry Tokens (EGTs), which encode detailed geometric structures extracted from reconstructed 3D attributes. A 3D-aware adapter fuses these two geometric representations with 2D visual cues. This pure-RGB design injects strong 3D inductive biases, enabling precise spatial understanding and reasoning without any additional 3D inputs such as depth sensors or LiDAR. Experiments show VLM-IE3D achieves strong performance across multiple 3D benchmarks, including 3D video grounding, 3D visual grounding, 3D dense captioning, and spatial reasoning. Paper: arXiv 2507.19321.

Paper Overview

Field: Computer Vision (CV) Authors: Wenhao Li, Xueying Jiang, Quanhao Qian arXiv: 2507.19321

Abstract (translated from the Chinese summary)

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs still struggle with 3D tasks that demand fine-grained spatial understanding and reasoning. To bridge this gap, the authors propose VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos.

Key components:

Implicit Geometry Tokens (IGTs): capture high-level geometric priors from the input videos.
Explicit Geometry Tokens (EGTs): encode detailed geometric structures extracted from reconstructed 3D attributes, complementing the IGTs.
3D-aware adapter: effectively fuses the two types of geometric representations with 2D visual cues.

This pure-RGB design injects strong 3D inductive biases, enabling fine-grained spatial understanding and reasoning without requiring any extra 3D input (no depth sensors or point clouds needed).

Results

Extensive experiments show that VLM-IE3D achieves superior performance on a variety of 3D tasks, including:

3D video detection
3D visual grounding
3D dense captioning
Spatial reasoning

Original Abstract (English)

> Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos.

Full paper: arXiv:2507.19321