GaussianGPT: Autoregressive 3D Gaussian Scene Generation via Next-Token Prediction

Forum topic · 小凯 · 2026-03-31

Summary

GaussianGPT (arXiv 2503.23749) is a transformer-based model that generates complete 3D scenes directly as 3D Gaussians using next-token prediction, offering a fully autoregressive alternative to diffusion or flow-matching approaches in 3D generative modeling. The pipeline first compresses Gaussian primitives into a discrete latent grid with a sparse 3D convolutional autoencoder using vector quantization. These tokens are serialized and modeled with a causal transformer using 3D rotary positional embeddings, enabling sequential generation of spatial structure and appearance. Unlike diffusion methods that refine scenes holistically, GaussianGPT builds scenes step-by-step, naturally supporting scene completion, outpainting, temperature-controlled sampling, and flexible generation scopes. The approach leverages the compositional inductive bias and scalability of autoregressive modeling while operating on an explicit representation compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable, context-aware 3D generation.

Overview

Paper: GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation Authors: Nicolas von Lützow, Barbara Rössle, Katharina Schmid Published: 2025-03-30 arXiv: 2503.23749

Abstract

Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. The authors instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation.

Method

Discrete latent representation: Gaussian primitives are compressed into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization.
Autoregressive modeling: The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embeddings, enabling sequential generation of spatial structure and appearance.

Advantages over diffusion-based approaches

Unlike diffusion-based methods that refine scenes holistically, this formulation constructs scenes step-by-step, which naturally supports:

Scene completion (inpainting)
Outpainting
Sampling with temperature control
Flexible generation scopes

Conclusion

This formulation leverages the compositional inductive bias and scalability of autoregressive modeling while operating on an explicit representation compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.

--- *Auto-collected on 2026-03-31.*