Paper Overview
Research area: Machine Learning Authors: Jiaming Zhang, Meng Ding, Shaopeng Fu, Jingfeng Zhang, Di Wang Published: 2026-04-21 arXiv: 2604.19724
Summary
Despite the remarkable success of Vision Transformers (ViTs) across a wide range of vision tasks, recent studies have revealed that they remain vulnerable to adversarial examples, much like Convolutional Neural Networks (CNNs). A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored.
In this work, the authors present the first theoretical analysis of adversarial training under simplified ViT architectures. They show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate perturbation budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes.
Remarkably, this leads to strong generalization even in the presence of overfitting—a phenomenon known as benign overfitting, previously observed only in adversarially trained CNNs. Experiments on synthetic and real-world datasets further validate the theoretical findings.
Key Takeaways
- First theoretical analysis of adversarial training for ViTs (under simplified architectures).
- Under suitable signal-to-noise ratio conditions and moderate perturbation budgets, adversarial training achieves nearly zero robust training loss and robust generalization error.
- Demonstrates benign overfitting in adversarially trained ViTs, extending a phenomenon previously only shown for CNNs.
- Empirical results on synthetic and real datasets corroborate the theory.
*Auto-collected on 2026-04-23.*