PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

Forum topic · 小凯 · 2026-07-08

Summary

PixWorld (arXiv:2607.05373) is a unified model that handles both 3D scene reconstruction and generation under a single pixel-space diffusion paradigm, proposed by Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, and Jia-Wang Bian. Traditionally, these tasks use separate approaches: pixel-based regression for reconstruction and latent diffusion for generation. Prior attempts to unify them in latent space suffer from drawbacks—the diffusion objective is defined on latent features rather than the underlying 3D representation, both branches lose information through latent encoding, and a pretrained VAE or RAE is required. PixWorld supervises diffusion directly on rendered images, aligning optimization with 3D scene fidelity and removing these limitations. Beyond photometric and perceptual supervision, which operate only at the 2D image level and lack 3D geometry awareness, the authors introduce a geometry-aware loss that aligns rendered views with ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision. PixWorld consistently outperforms previous latent-space generation methods and matches state-of-the-art reconstruction approaches.

Paper Overview

Field: Computer Vision
Authors: Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Jia-Wang Bian
Published: 2026-07-06
arXiv: 2607.05373

Abstract

3D reconstruction and generation are typically handled by different paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent work attempts to unify them in latent space, but suffers from notable drawbacks:

The diffusion objective is defined on latent features rather than the underlying 3D representation.
Both branches incur information loss from latent encoding.
A pretrained VAE or RAE is required.

This paper re-unifies the two tasks under a pixel-space diffusion paradigm, proposing PixWorld—a single model that jointly handles 3D reconstruction and generation.

Method Highlights

By supervising diffusion directly on rendered images, PixWorld eliminates the above limitations and aligns optimization with 3D scene fidelity.
In addition to photometric and perceptual supervision—which operate only at the 2D image level and lack 3D geometry awareness—the authors introduce a geometry-aware loss that aligns rendered views with ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision.

Results

PixWorld consistently outperforms previous latent-space generation methods and matches state-of-the-art reconstruction methods.

*Auto-collected on 2026-07-06.*