Overview
Field: Computer Vision Authors: Xingjian Leng, Jaskirat Singh, Zhanhao Liang Published: 2026-06-24 arXiv: 2506.14783
Summary
Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While various methods improve FID and related metrics, it is increasingly unclear whether these improvements reflect real progress in generative modeling. The natural alternative — text-to-image (T2I) generation — is often skipped because it is perceived as too costly to train or inconvenient to evaluate. This paper argues that this perception no longer holds.
The authors introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with only 12 lines of configuration changes, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I settings. Under NanoGen, the compute required to train T2I models is comparable to ImageNet.
Key Findings
- After training 21 latent diffusion models with NanoGen, the authors observe no strong correlation between method rankings on ImageNet and T2I generation: Pearson correlation coefficients across three metrics range from -0.377 to -0.580.
- This means methods that improve class-conditional FID on ImageNet may show no corresponding improvement on T2I, clearly pointing to the necessity of evaluating DiTs on both tasks.
DiffusionBench
To address this, the authors aggregate ImageNet and text-to-image results into DiffusionBench, a holistic benchmark for DiT research. They recommend replacing standalone ImageNet reporting with DiffusionBench: methods that improve on DiffusionBench are more likely to reflect broader progress in generative modeling.
--- *Auto-collected on 2026-06-25*