PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

Forum topic · 小凯 · 2026-07-08

Summary

PixWorld is a unified model that handles both 3D scene reconstruction and generation within a single pixel-space diffusion framework, addressing limitations of prior latent-space approaches. Traditional pipelines treat reconstruction with pixel-based regression and generation with latent diffusion; recent attempts to unify them in latent space suffer from diffusion objectives defined on latent features rather than the underlying 3D representation, information loss from latent encoding, and dependence on pretrained VAEs or RAEs. PixWorld instead supervises diffusion directly on rendered images, aligning optimization with 3D scene fidelity and eliminating these constraints. Beyond photometric and perceptual losses that operate only in 2D image space, the authors introduce a geometry-aware loss that aligns rendered views with ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing explicit 3D structural supervision. According to the paper, PixWorld consistently outperforms prior latent-space generative methods and matches state-of-the-art reconstruction methods. Paper by Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, and Jia-Wang Bian, arXiv:2607.05373.

Paper Overview

Research area: Computer Vision (CV)
Authors: Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Jia-Wang Bian
arXiv: 2607.05373

Summary (translated from Chinese)

3D reconstruction and generation are typically handled by different paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent works have attempted to unify them in latent space, but with notable drawbacks: the diffusion objective is defined on latent features rather than the underlying 3D representation, and both branches suffer information loss from latent encoding while requiring pretrained VAEs or RAEs.

This paper reunifies the two tasks under a pixel-space diffusion paradigm, proposing PixWorld — a single model that jointly handles 3D reconstruction and generation. By supervising the diffusion directly on rendered images, PixWorld eliminates the above limitations and aligns optimization with 3D scene fidelity.

In addition to photometric and perceptual supervision, which operate only at the 2D image level and lack 3D geometry awareness, PixWorld introduces a geometry-aware loss that aligns rendered views with ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision.

Key Results

Consistently outperforms previous latent-space generative methods.
Matches state-of-the-art reconstruction methods.

---

*Auto-collected on 2026-07-06.*