GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

Forum topic · 小凯 · 2026-09-23

Summary

GAE (Geometry-Native Autoencoder) is a compact latent space designed as a shared foundation for perception and generation in 3D visual synthesis. The authors argue that inconsistent 3D scenes in visual generators stem not only from modeling but from representation: generators evolve appearance-centric latents, while perception models encode cross-view geometric structure. GAE reparameterizes features from a geometry foundation model into a compact latent space that can be jointly decoded into appearance, depth, camera poses, and point maps. With this state, a standard conditional flow model supports diverse generation tasks. In controlled comparisons with fixed generators and training protocols, swapping the latent space to GAE improved both visual quality and measured 3D consistency: FVD dropped 12.7% on RealEstate10K and 23.1% on DL3DV, and camera trajectory error on RealEstate10K was halved. The results suggest the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.

Paper Overview

Field: Computer Vision (CV) Authors: Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu Published: 2026-09-21 arXiv: 2609.24981

Introduction

Visual generators can produce photorealistic frames, but they often fail to preserve a consistent 3D scene. The authors argue this is not only a modeling problem but also a representation problem:

Generators typically evolve appearance-centric latents.
Perception models recover geometry in a semantically rich space that encodes cross-view structure.

Rather than adding geometry as another output, the paper proposes reparameterizing a geometry foundation model's features into a compact latent space for generation.

Method: Geometry-Native Autoencoder (GAE)

GAE implements this shift with a latent space that is jointly decodable into:

Appearance (RGB frames)
Depth
Camera poses
Point maps

With this state, a standard conditional flow model supports diverse generation tasks without task-specific architectural changes.

Results

In controlled comparisons with fixed generators and identical training protocols, replacing the latent space with GAE improved both visual quality and independently measured 3D consistency:

FVD decreased 12.7% on RealEstate10K
FVD decreased 23.1% on DL3DV
Camera trajectory error halved on RealEstate10K

Conclusion

These results indicate that the latent space itself is central to geometry-consistent generation, and that a geometry-native latent space can serve as a shared interface between perception and generation.

*Auto-collected on 2026-09-23.*