Z3D: Zero-Shot Novel Depth Synthesis Using 3D Foundation Model Scene Representations

Forum topic · 小凯 · 2026-09-07

Summary

This paper (arXiv:2509.04284) by Denis M. Akola and David F. Fouhey explores how 3D foundation models (3DFMs) such as VGGT can infer 3D scene information from novel views. The authors hypothesize that to solve 3D reconstruction, these feed-forward transformers must learn representations containing substantial general knowledge about 3D scenes. They first demonstrate that hidden surfaces can be decoded from the internal representations of 3DFMs, then propose Z3D, a method that estimates pointmaps for unseen views by performing latent diffusion over the 3DFM representation. Experiments show Z3D can predict realistic depth maps for novel viewpoints across multiple datasets, enabling zero-shot novel view depth synthesis without task-specific training. The work bridges 3D foundation model representations and generative diffusion techniques for view extrapolation.

Paper Overview

Field: Computer Vision Authors: Denis M. Akola, David F. Fouhey Published: 2026-09-06 arXiv: 2509.04284

Abstract

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-forward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks.

In this paper, the authors investigate using these internal representations to infer 3D information in the scene from new views. Their hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes.

Method: Z3D

After showing that it is possible to decode hidden surfaces from internal 3DFM representations, the authors propose Z3D, a method that estimates pointmaps in unseen views by performing latent diffusion on the 3DFM representation.

Results

Hidden surfaces can be decoded from internal 3DFM representations, supporting the hypothesis that these models encode broad 3D scene knowledge.
Z3D can predict realistic depth maps for novel viewpoints across multiple datasets, enabling zero-shot novel depth synthesis.