Overview
Research Area: Computer Vision Authors: Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao, Pengfei Wan, Kun Gai, Li Li, Houqiang Li Published: 2026-10-13 arXiv: 2610.18857
Summary
Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. This paper demonstrates that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion.
Building on this finding, the authors introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE — requiring no diffusion-specific routing mechanisms or auxiliary routing objectives. Key design elements:
- Token-Choice (TC) routing combined with fine-grained routed experts for input-adaptive sparse computation
- Shared experts to capture common representations
- Bias-based load balancing strategy
- At comparable active parameter counts, PixelMoE consistently outperforms dense counterparts across model scales, demonstrating effective sparse capacity scaling for pixel-space generation.
- On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256x256 and 1.53 at 512x512 — without representation alignment or perceptual losses — remaining competitive with leading pixel-space and latent-space generative models.
- Paper: https://arxiv.org/abs/2610.18857
To complement sparse capacity scaling, PixelMoE introduces hierarchical flow matching in pixel space, varying spatial resolution along the denoising trajectory — progressing from global structure formation to fine-grained detail refinement.