Decoding Register vs. High-Norm Outlier Patch Tokens in DINOv2 with Sparse Autoencoders

Forum topic · 小凯 · 2026-10-06

Summary

Self-supervised Vision Transformers like DINOv2 learn rich visual representations, but the internal roles of their tokens remain poorly understood. This paper (arXiv:2610.03698) analyzes two token types: dedicated register tokens and the high-norm outlier patch tokens that tend to appear in background regions. The authors train sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2, then interpret the learned features via an automated interpretability pipeline, UMAP clustering, and CLIP-space cross-checks. They find register-token features are more strongly associated with high-level semantic concepts, while outlier-token features relate more to low-level structure, background, and texture-dominated patterns. Causal ablation experiments reveal a striking functional asymmetry: destroying the strongest register-derived features causes a 48.17% drop in representation cosine similarity, whereas destroying outlier-derived features causes only a 0.31% drop. The results provide evidence of functional token specialization in self-supervised ViTs.

Overview

Field: Computer Vision Authors: Neel Varma, Andrew Rufail, Dipika Khullar, Vasu Sharma Published: 2026-10-02 arXiv: 2610.03698

Summary

Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly understood. Recent architectures introduce dedicated register tokens to reduce high-norm outlier patch tokens that emerge in background regions, yet the semantic and functional roles of both token types have not been fully established.

Method

The authors analyze these roles by training sparse autoencoders (SAEs) on register-token and outlier-token activations in DINOv2. Interpretation uses:

An automated interpretability pipeline
UMAP clustering
CLIP-space cross-checks

Findings

Register-token features are more strongly associated with high-level semantic concepts.
Outlier-token features are more often associated with low-level structure, background, and texture-dominated patterns.
Causal ablation reveals a significant functional asymmetry: destroying the most strongly activating register-derived features causes a 48.17% drop in representation cosine similarity, while destroying outlier-derived features causes only a 0.31% drop.

Conclusion

These results provide evidence of functional token specialization in self-supervised Vision Transformers.

--- *Auto-collected on 2026-10-06*

Decoding Register vs. High-Norm Outlier Patch Tokens in DINOv2 with Sparse Autoencoders

Summary

Overview

Summary

Method

Findings

Conclusion

Tags