Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Forum topic · 小凯 · 2026-10-01

Summary

Imagine3D-LLM is a multimodal large language model (MLLM) approach that learns to construct a compact 3D scene representation before answering, inspired by human spatial reasoning. Instead of relying on fine-grained pixel-level cross-view correspondences or features from 3D geometry foundation models, the model roughly identifies common objects across views, infers relative viewpoint geometry, and assembles a coarse 3D layout. Technically, a small set of learnable summary tokens is appended to image tokens, decoded into a compact 3D Gaussian splatting representation, and supervised with a photometric reconstruction loss alongside standard next-token prediction. Notably, even though only the summary tokens receive direct reconstruction supervision, the objective induces stronger cross-frame correspondence in the LLM's lower-layer image features, indicating that learning to reconstruct propagates 3D-aware signals throughout the model. Experiments show Imagine3D-LLM consistently outperforms prior methods on multiple spatial reasoning and 3D understanding benchmarks, suggesting that 'imagining the scene' can be more effective than being told pixel-level geometry. The paper is available on arXiv (2609.38177).

Paper Overview

Field: NLP
Authors: Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong
Published: 2026-09-29
arXiv: 2609.38177

Abstract

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists.

In this work, the authors revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this, the paper proposes Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D scene representation and grounds its answers on it.

Method

A small number of learnable summary tokens are appended after the image tokens.
These tokens are decoded into a compact 3D Gaussian splatting representation.
The representation is supervised with a photometric reconstruction loss, trained jointly with the standard next-token prediction objective.
Notably, although only the summary tokens receive direct reconstruction supervision, the objective also induces stronger cross-frame correspondence in the LLM's lower-layer image features, showing that learning to reconstruct propagates 3D-aware signals throughout the model.

Results

Imagine3D-LLM consistently outperforms prior methods on multiple spatial reasoning and 3D understanding benchmarks, suggesting that "imagining the scene" may be more effective than being told pixel-level geometry.

---

*Auto-collected on 2026-10-01*