Pixel-space diffusion models skip the VAE reconstruction bottleneck—instead of compressing images into a latent space before denoising, they operate directly in raw pixel space. That's a win: no compression loss. But it introduces a new problem:
If you want to capture global semantics, you should use large patches—say, grouping 4×4 or even 8×8 pixels—processing large regions at once. But if you want to generate fine details, you need small patches—1×1 or finer. You can't have both at once.
HyperDiT (arXiv:2605.15741) takes a direct approach to this problem: why not use both? It lets fine-grained patch tokens query global information through cross-attention from multi-level semantic anchors. It's like giving every pixel its own magnifying glass—seeing its own local texture while knowing the global context of "this is a cat's face."
Key design choices
1. Cross-attention for semantic injection instead of the conventional AdaLN. 2. Scale-aware rotary position embeddings (SA-RoPE) to guarantee precise geometric alignment between different patch sizes. 3. Dense semantics from register tokens learned from pretrained vision foundation models, reducing hallucination.
Results
State-of-the-art FID of 1.56 on ImageNet 256×256.
Open questions
- How much computational overhead does cross-attention add compared to AdaLN? Pixel-space operation is already more expensive than latent-space; adding cross-scale cross-attention could further increase cost.
- The paper reports FID, but no inference-speed comparison is provided.
References
1. He, Y., et al. (2026). *HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion*. arXiv:2605.15741 [cs.CV]. 2. Peebles, W., & Xie, S. (2023). *Scalable Diffusion Models with Transformers*. ICCV 2023. 3. Ho, J., et al. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. 4. Chen, J., et al. (2025). *Pixel-Space Diffusion Models: A New Paradigm for Image Generation*.