Paper: 3D-Aware VLMs with Implicit and Explicit Geometries (VLM-IE3D)

Forum topic · 小凯 · 2026-07-26

Summary

VLM-IE3D is a unified framework that improves the 3D spatial awareness of vision-language models using only RGB video input. It introduces Implicit Geometry Tokens (IGTs), which capture high-level geometric priors from input videos, and complementary Explicit Geometry Tokens (EGTs), which encode detailed geometric structures from reconstructed 3D attributes. A 3D-aware adapter fuses both geometric representations with 2D visual cues, injecting strong 3D inductive biases without requiring any additional 3D inputs such as depth sensors or point clouds. Experiments show consistent state-of-the-art performance across multiple 3D benchmarks, including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. The work, authored by Wenhao Li, Xueying Jiang, and Quanhao Qian, is available on arXiv (2507.20491), with code and models released at github.com/Vegetebird/VLM-IE3D. This post is an auto-collected paper summary from zhichai.net covering the abstract and paper metadata.

Paper Overview

Field: Computer Vision (CV) Authors: Wenhao Li, Xueying Jiang, Quanhao Qian Published: 2026-07-25 arXiv: 2507.20491

Chinese Abstract (translated)

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, the authors present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos.

VLM-IE3D introduces:

Implicit Geometry Tokens (IGTs) — capture high-level geometric priors from input videos
Explicit Geometry Tokens (EGTs) — encode detailed geometric structures from reconstructed 3D attributes
3D-aware adapter — effectively fuses the two types of geometric representations with 2D visual cues

This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning, without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves consistently strong performance across multiple 3D tasks, including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning.

Resources

Code and models: https://github.com/Vegetebird/VLM-IE3D
Paper: https://arxiv.org/abs/2507.20491

---

*Auto-collected on 2026-07-26.*