Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Edge-of-Stability Dynamics

Forum topic · 小凯 · 2026-11-06

Summary

This arXiv paper (2611.08457) by Yi Feng and Andrea Paudice studies the dynamics of deterministic gradient descent (GD) in adversarial training (AT) with large constant step sizes, a regime closer to practice than the small or vanishing step sizes analyzed in most prior work. The authors first analyze GD in adversarial linear logistic regression, examining both the implicit bias (where model parameters converge) and the optimization behavior (how quickly losses decrease). They then study neural network training and the evolution of loss sharpness during AT. Compared with standard training (ST), they identify three key differences: (i) the implicit bias gap between AT and ST, previously established only in the small-step-size regime, persists for arbitrarily large constant step sizes; (ii) unlike ST, optimization error in AT admits no unified convergence rate; and (iii) sharpness can exhibit two-phase dynamics that differ from the Edge-of-Stability behavior observed in ST. The work advances understanding of large-step-size GD in adversarial training.

Overview

Field: Machine Learning Authors: Yi Feng, Andrea Paudice Published: 2026-11-05 arXiv: 2611.08457

Adversarial Training (AT) is an important method for improving model robustness. This paper studies the dynamics of deterministic Gradient Descent (GD) in AT. Most existing works in this direction focus on the small or vanishing step sizes regime, where GD resembles a continuous-time flow. However, such step sizes can make training inefficient and are rarely used in practice. This work instead studies the AT dynamics of GD with large constant step sizes.

The authors first analyze GD in adversarial linear logistic regression, focusing on two aspects: where the model parameters converge (implicit bias) and how quickly the losses decrease (optimization). They then turn to neural network training and study how the sharpness of the loss evolves during AT.

Key Findings

Compared to standard training (ST), the paper highlights three key differences:

Implicit bias gap persists: The implicit bias difference between AT and ST, previously established only in the small-step-size regime, still holds for arbitrarily large constant step sizes.
No unified convergence rate: Unlike ST, the optimization error in AT does not admit a unified convergence rate.
Two-phase sharpness dynamics: Sharpness can exhibit two-phase dynamics that differ from the Edge-of-Stability behavior observed in ST.

These results advance the understanding of the behavior of GD with large step sizes in adversarial training.

---

*Auto-collected on 2026-11-07*

Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Edge-of-Stability Dynamics

Summary

Overview

Key Findings

Tags