Z3D: Zero-Shot Novel Depth Synthesis Using 3D Foundation Model Scene Representations

Forum topic · 小凯 · 2026-09-07

Summary

This paper by Denis M. Akola and David F. Fouhey (arXiv:2509.04284) explores whether the internal representations of 3D foundation models (3DFMs) such as VGGT encode generalizable knowledge about 3D scenes. The authors hypothesize that to solve 3D reconstruction, these feed-forward transformers must learn representations containing broad general knowledge about 3D scenes. They first demonstrate that hidden surfaces can be decoded from a 3DFM's internal representations, then propose Z3D, a method that performs latent diffusion over 3DFM representations to estimate pointmaps for unseen viewpoints. Experiments show Z3D can predict photorealistic depth maps for novel views across multiple datasets in a zero-shot manner. The work suggests 3D foundation models serve as rich scene priors beyond their original training tasks, enabling novel-view depth synthesis without view-specific training.

Paper Overview

Research Area: Computer Vision (CV) Authors: Denis M. Akola, David F. Fouhey arXiv: 2509.04284

Abstract

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-forward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, the authors investigate using their internal representations to infer 3D information in the scene from new views.

The central hypothesis is that, in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes.

Method: Z3D

1. Decoding hidden surfaces: The authors first show that it is possible to decode hidden surfaces from internal 3DFM representations, suggesting these features capture more than what is directly observed. 2. Latent diffusion on 3DFM features: Building on this finding, they propose Z3D, which estimates pointmaps in unseen views by performing latent diffusion over the 3DFM representation.

Results

Z3D can predict photorealistic depth maps for novel views across multiple datasets, demonstrating zero-shot novel depth synthesis powered by 3D foundation model scene representations.

---

*Auto-collected on 2026-09-07.*