GeoNeXt: Video Generative Models as Geometry Learners for Unified Depth and Normal Estimation

Forum topic · 小凯 · 2026-09-01

Summary

GeoNeXt (arXiv:2608.28549) repurposes pretrained video generative models as a unified, data-efficient framework for monocular geometry estimation, formulated as a next-frames prediction task. Existing generative geometry approaches adapt image diffusion models either by training task-specific models for depth and surface normal estimation independently—missing correlations between geometric targets—or by jointly fine-tuning modified diffusion backbones, which requires large labeled datasets. GeoNeXt instead inherits the naturally structured knowledge and richer priors of video generation models, then adapts them to jointly model image and geometry targets (image-geometry), enabling more efficient and effective geometric learning. Extensive experiments demonstrate strong zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming prior task-specific and unified generative competitors while using substantially less training data. Notably, GeoNeXt rivals state-of-the-art discriminative methods trained with over 100x more data and even surpasses them on several benchmarks. Authored by Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, and Jiankang Deng, the work is positioned in computer vision.

Paper Overview

Field: Computer Vision Authors: Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng arXiv: 2608.28549

Abstract

Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data.

To overcome these limitations in a principled fashion, the authors repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Their method, GeoNeXt, inherits the naturally structured knowledge and richer priors of video models, while further adapting them to jointly model image and geometry targets (image-geometry), enabling more efficient and effective geometry learning.

Key Results

Extensive experiments validate strong zero-shot monocular depth and surface normal estimation across diverse datasets.
Outperforms previous task-specific and unified generative competitors while using substantially less training data.
Comparable to state-of-the-art discriminative methods trained with over 100x more data, even outperforming them on several benchmarks.

Discussion

Share your thoughts on video-generation-based geometry estimation below. Does next-frames prediction naturally suit joint depth/normal modeling?

--- *Auto-collected on 2026-09-01*