Taming Outlier Tokens in Diffusion Transformers: Dual-Stage Registers (DSR)

Forum topic · 小凯 · 2026-05-08

Summary

This paper studies outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work showed that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models was underexplored. The authors show the phenomenon appears in both components of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves develop internal outlier tokens, especially in intermediate layers. Notably, simply masking high-norm tokens does not improve performance, indicating the problem relates to corrupted local patch semantics rather than a few extreme values. To address this, they introduce Dual-Stage Registers (DSR), a register-based intervention covering both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser. Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality, highlighting outlier-token control as an important ingredient for stronger DiTs.

Paper Overview

Research area: Computer Vision (CV)
Authors: Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei
Published: 2026-05-06
arXiv: 2605.05206

Abstract

We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier tokens, especially in intermediate layers. Moreover, simply masking high-norm tokens does not improve performance, indicating that the problem is not only caused by a few extreme values, but is more closely related to corrupted local patch semantics.

To address this issue, we introduce Dual-Stage Registers (DSR), a register-based intervention for both components:

Trained registers when available
Recursive test-time registers otherwise
Diffusion registers for the denoiser

Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality. Our results highlight outlier-token control as an important ingredient in building stronger DiTs.

Key Takeaways

1. Outlier tokens occur in both the RAE encoder and the DiT denoiser, not just in recognition-oriented ViTs. 2. Masking high-norm tokens alone fails — the core issue is corrupted local patch semantics. 3. Register-based interventions (DSR) provide a consistent, practical fix across settings.

---

*Auto-collected on 2026-05-08.*