Pixel-space diffusion models skip the VAE reconstruction bottleneck — instead of compressing images into a latent space before denoising, they operate directly in the original pixel space. That's a plus: no compression loss. But it introduces a new problem:
If you want to capture global semantics, large patches work best — for example, grouping 4×4 or even 8×8 pixels and processing large regions at once. But if you want to generate fine details, you need small patches — 1×1 or even finer. You can't have both at the same time.
HyperDiT (arXiv:2605.15741) takes a direct approach to this dilemma: why not use both? It lets fine-grained patch tokens query global information from multi-level semantic anchors via cross-attention. It's like giving every pixel its own "magnifying glass" — each one sees its local texture while also knowing the global context, e.g., "this is a cat's face."
Key design elements
1. Cross-attention for semantic injection instead of the conventional AdaLN mechanism. 2. Scale-aware rotary position embeddings (SA-RoPE) to guarantee precise geometric alignment between different patch sizes. 3. Dense semantics from registered tokens of pretrained vision foundation models, reducing hallucination.
Results
State-of-the-art FID of 1.56 on ImageNet 256×256.
Open questions
- How much computational overhead does the cross-attention mechanism add compared to AdaLN? Pixel-space operation is already more expensive than latent-space methods, and adding cross-scale cross-attention could further increase cost.
- The paper reports FID, but no inference speed comparison is provided.
References
1. He, Y., et al. (2026). *HyperDiT: Hyper-Connected Transformers for High-Fidelity Pixel-Space Diffusion*. arXiv:2605.15741 [cs.CV]. 2. Peebles, W., & Xie, S. (2023). *Scalable Diffusion Models with Transformers*. ICCV 2023. 3. Ho, J., et al. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. 4. Chen, J., et al. (2025). *Pixel-Space Diffusion Models: A New Paradigm for Image Generation*.