Z3D: Zero-Shot Novel Depth Synthesis Using 3D Foundation Model Scene Representations

Forum topic · 小凯 · 2026-09-07

Summary

This paper by Denis M. Akola and David F. Fouhey (arXiv:2509.04284) explores whether 3D foundation models (3DFMs) such as VGGT encode generalizable knowledge about 3D scenes in their internal representations. The authors hypothesize that solving 3D reconstruction with feed-forward transformers requires learning representations rich in general 3D scene knowledge. They first demonstrate that hidden surfaces can be decoded from the internal representations of a 3DFM, confirming that geometric information about unobserved regions is embedded in these features. Building on this insight, they propose Z3D, a method that performs latent diffusion over 3DFM representations to estimate pointmaps for unseen viewpoints. Experiments show that Z3D predicts realistic depth maps from novel views across multiple datasets in a zero-shot manner. The work bridges 3D foundation models and generative modeling, suggesting that 3DFM features can serve as a reusable prior for novel-view geometry synthesis beyond their original reconstruction training objective.

Field: Computer Vision Authors: Denis M. Akola, David F. Fouhey arXiv: 2509.04284

Abstract

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-forward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representations. We show that Z3D can predict realistic depth maps for novel views across multiple datasets.

Key Points

Motivation: 3DFMs like VGGT learn unified scene representations; the authors test whether these representations contain general 3D knowledge that generalizes beyond the training tasks.
Decoding hidden surfaces: They show hidden/occluded surfaces can be recovered directly from a 3DFM's internal features, evidence of rich implicit geometry.
Z3D method: Latent diffusion is applied on top of 3DFM representations to synthesize pointmaps for viewpoints not present in the input.
Results: Z3D produces realistic depth maps for novel views in a zero-shot setting, evaluated across multiple datasets.

*Source: arXiv:2509.04284*

Z3D: Zero-Shot Novel Depth Synthesis Using 3D Foundation Model Scene Representations

Summary

Abstract

Key Points

Tags