Paper Overview
Field: Computer Vision Authors: Xingjian Leng, Jaskirat Singh, Zhanhao Liang Published: 2026-06-24 arXiv: 2506.14783
Summary
Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While various methods improve FID and related metrics, it is increasingly unclear whether these improvements reflect genuine progress in generative modeling. The natural alternative—text-to-image (T2I) generation—is often skipped because it is perceived as too costly to train or inconvenient to evaluate. This paper argues that this perception no longer holds.
The authors propose NanoGen, a unified DiT training and evaluation framework. NanoGen:
- Matches state-of-the-art DiT baselines on ImageNet
- Trains competitive text-to-image models with only 12 lines of configuration changes
- Supports RAE, VAE, pixel-space, and MeanFlow diffusion methods
- Works in both ImageNet and T2I settings
- Pearson correlation coefficients across three metrics ranged from -0.377 to -0.580
- Methods that improve class-conditional FID on ImageNet may not improve T2I results correspondingly
- This clearly demonstrates the necessity of evaluating DiTs on both tasks
Under NanoGen, the compute required to train a T2I model is comparable to ImageNet class-conditional training.
Key Findings
After training 21 latent diffusion models with NanoGen, the authors observed no strong correlation between method rankings on ImageNet and T2I generation:
DiffusionBench
To address this gap, the authors aggregate ImageNet and text-to-image results into DiffusionBench, a comprehensive benchmark for DiT research. They recommend replacing standalone ImageNet reporting with DiffusionBench: methods that improve on DiffusionBench are more likely to reflect broader progress in generative modeling.
---
*Automatically collected on 2026-06-25.*