Training Pixel-Space Text-to-Image Diffusion Models: A Latent-to-Pixel Empirical Study (arXiv 2608.16887)

Forum topic · 小凯 · 2026-08-19

Summary

A new empirical study (arXiv 2608.16887) examines how to train pixel-space text-to-image diffusion models that rival established latent-space models. The authors, led by Dengyang Jiang, Ruoyi Du, and Zhennan Chen, observe that direct large-scale pre-training in pixel space converges substantially slower than in latent space. Based on this finding, they propose a latent-to-pixel strategy: efficiently acquire generative priors via latent-space pre-training, then transition to pixel space during post-training. The paper systematically investigates the key design choices governing this transition, including weight initialization, data composition, prediction objective, decoder architecture, and noise schedules. The resulting recipe enables pixel-space models that match or exceed their latent-space counterparts while achieving 3.18x to 4.75x end-to-end inference acceleration. The work addresses a gap, since most prior pixel-space diffusion research was limited to small-scale or class-conditional settings, and offers a practical guideline for scaling pixel-space generation models.

Paper Overview

Research area: Computer Vision (CV)
Authors: Dengyang Jiang, Ruoyi Du, Zhennan Chen et al. (13 authors)
Published: 2026-08-17
arXiv: 2608.16887

Key Points

Studies pixel-space diffusion models for text-to-image generation, a topic where most existing work is limited to small-scale or class-conditional settings.
Empirically shows that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space.
Proposes a latent-to-pixel strategy: first acquire generative priors efficiently in latent space, then transition to pixel space during post-training.
Systematically investigates the key design choices for this transition:
Weight initialization
Data composition
Prediction objective
Decoder architecture
Noise schedules
The resulting practical recipe produces pixel-space models that match or exceed their latent-space counterparts.
Achieves 3.18x to 4.75x end-to-end inference acceleration compared to latent-space baselines.

Abstract (Original)

This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction objective, decoder architecture, and noise schedules, and identify a practical recipe that enables the resulting pixel-space models to match or exceed their latent-space counterparts while achieving 3.18x–4.75x end-to-end inference acceleration.

---

*Auto-collected on 2026-08-19.*