GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

Forum topic · 小凯 · 2026-09-23

Summary

GAE (Geometry-Native Autoencoder) introduces a compact geometry-native latent space designed as a shared foundation for perception and generation. The authors argue that the failure of visual generators to maintain consistent 3D scenes is fundamentally a representation problem: generators typically learn appearance-centric latents, while perception models recover geometry in spaces rich with cross-view structure. Instead of adding geometry as an extra output, GAE reparameterizes the features of a geometry foundation model into a compact generative latent space that is jointly decodable to appearance, depth, camera poses, and point maps. With this representation, a standard conditional flow model supports diverse generation tasks. In controlled comparisons with a fixed generator and training protocol, swapping the latent space for GAE improved both visual quality and independently measured 3D consistency: FVD dropped by 12.7% on RealEstate10K and 23.1% on DL3DV, and camera trajectory error on RealEstate10K was halved. The results indicate that the latent space itself is central to geometry-consistent generation and can serve as a shared interface between perception and generation. Paper: arXiv 2609.24981.

Paper Overview

Field: Computer Vision Authors: Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu Published: 2026-09-21 arXiv: 2609.24981

Abstract

We present a compact geometry-native latent space as a shared foundation for perception and generation. Visual generators can produce photorealistic frames without preserving a consistent 3D scene. We argue that this is not only a modeling problem but also a representation problem: generators typically evolve appearance-centric latents, while perception models recover geometry in a semantically rich space that encodes cross-view structure. Rather than adding geometry as another output, we reparameterize a geometry foundation model's features into a compact latent space for generation.

We realize this shift with the geometry-native autoencoder (GAE), whose latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks.

Key Results

In controlled comparisons with a fixed generator and training protocol, replacing the latent space with GAE improves both visual quality and independently measured 3D consistency:

FVD reduced by 12.7% on RealEstate10K
FVD reduced by 23.1% on DL3DV
Camera trajectory error halved on RealEstate10K

These results show that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.

--- *Auto-collected on 2026-09-23*