Benign Overfitting in Adversarial Training for Vision Transformers

Forum topic · 小凯 · 2026-04-23

Summary

This paper provides the first theoretical analysis of adversarial training for Vision Transformers (ViTs), which are known to be vulnerable to adversarial examples similar to CNNs. Under simplified ViT architectures, the authors prove that adversarial training can achieve nearly zero robust training loss and robust generalization error when the signal-to-noise ratio satisfies a specific condition and the perturbation budget is moderate. A key finding is the occurrence of benign overfitting in ViTs: strong generalization despite overfitting, a phenomenon previously observed only in adversarially trained CNNs. Experiments on synthetic and real-world datasets validate the theoretical results. The work bridges a gap in understanding the robustness foundations of ViTs and extends the benign overfitting phenomenon beyond CNNs.

Paper Overview

Research Area: Machine Learning
Authors: Jiaming Zhang, Meng Ding, Shaopeng Fu, Jingfeng Zhang, Di Wang
Published: 2026-04-21
arXiv: 2604.19724

Abstract (Chinese summary translated)

Although Vision Transformers (ViTs) have achieved remarkable success across a wide range of vision tasks, recent studies have shown that, much like Convolutional Neural Networks (CNNs), they remain vulnerable to adversarial examples. Adversarial training is a common empirical defense strategy, but its theoretical robustness guarantees remain largely unexplored for ViTs.

In this work, the authors present the first analysis of adversarial training under simplified ViT architectures. They prove that, when the signal-to-noise ratio satisfies a specific condition and the perturbation budget is moderate, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes. Remarkably, this leads to strong generalization even when overfitting is present — a phenomenon known as benign overfitting, previously observed only in adversarially trained CNNs.

Experiments on synthetic and real-world datasets further validate the theoretical findings.

Original Abstract

Despite the remarkable success of Vision Transformers (ViTs) across a wide range of vision tasks, recent studies have revealed that they remain vulnerable to adversarial examples, much like Convolutional Neural Networks (CNNs). A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored. In this work, we present the first theoretical analysis of adversarial training under simplified ViT architectures. We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate perturbation budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes. Remarkably, this leads to strong generalization even in the presence of overfitting — known as "benign overfitting," previously observed only in (adversarially trained) CNNs.

Key Points

First theoretical analysis of adversarial training for simplified ViT architectures.
Identifies conditions on signal-to-noise ratio and perturbation budget under which adversarial training achieves near-zero robust training loss and robust generalization error.
Demonstrates the benign overfitting phenomenon in ViTs, extending it beyond the previously studied CNN setting.
Empirical validation on both synthetic and real-world datasets supports the theory.
Bridges robustness theory and practical adversarial defenses for transformer-based vision models.

--- *Auto-collected on 2026-04-23*