GaussianGPT: Autoregressive Transformer for 3D Gaussian Scene Generation

Forum topic · 小凯 · 2026-03-31

Summary

GaussianGPT is a fully autoregressive alternative to diffusion- and flow-matching-based 3D generative models. The transformer-based system directly generates 3D Gaussians via next-token prediction, enabling complete 3D scene generation. Gaussian primitives are first compressed into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled with a causal transformer equipped with 3D rotary positional embeddings, allowing sequential generation of spatial structure and appearance. Unlike diffusion approaches that refine scenes holistically, GaussianGPT builds scenes step-by-step, naturally supporting completion, outpainting, temperature-controlled sampling, and flexible generation scopes. The method leverages the compositional inductive bias and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines. Authored by Nicolas von Lützow, Barbara Rössle, and Katharina Schmid, the paper was released on arXiv (2503.23749) on March 30, 2025, positioning autoregressive transformers as a complementary paradigm for controllable, context-aware 3D generation.

Paper Overview

Field: Computer Vision
Authors: Nicolas von Lützow, Barbara Rössle, Katharina Schmid
Published: 2025-03-30
arXiv: 2503.23749

Abstract

Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. The authors instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation.

Method

1. Discrete latent representation: Gaussian primitives are compressed into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. 2. Autoregressive modeling: The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embeddings (RoPE), enabling sequential generation of spatial structure and appearance.

Key Properties

Unlike diffusion-based methods that refine scenes holistically, GaussianGPT constructs scenes step-by-step.
This formulation naturally supports:
Scene completion
Outpainting
Temperature-controlled sampling
Flexible generation scopes
The approach leverages the compositional inductive bias and scalability of autoregressive modeling, while operating on explicit representations compatible with modern neural rendering pipelines.

Conclusion

The work positions autoregressive transformers as a complementary paradigm to diffusion methods for controllable and context-aware 3D scene generation.

---

*Auto-collected on 2026-03-31.*