Sharpness-Aware Minimization and Muon: Robustness under Spectral-Norm Geometry

Forum topic · 小凯 · 2026-07-30

Summary

This paper (arXiv:2607.26001) by Wenzhi Zhong, Edward Milsom, and Michael Murray studies how the choice of perturbation geometry affects Sharpness-Aware Minimization (SAM), a technique that improves generalization by encouraging insensitivity to small worst-case parameter perturbations. Motivated by matrix-aware optimizers such as Muon, which show that respecting the matrix structure of hidden-layer weights yields strong empirical performance, the authors apply matrix-aware geometry to both stages of SAM. They introduce a layerwise spectral-norm inner perturbation for matrix-valued hidden-layer parameters and combine it with AdamW/SGDW or Muon for the outer update. Experiments training ViT-Small/16 and ResNet-50 on ImageNet-1K show that the combination of spectral-norm inner steps with Muon outer steps performs consistently strongly, achieving the best validation accuracy among the evaluated methods on both architectures.

Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm

Research area: Machine Learning Authors: Wenzhi Zhong, Edward Milsom, Michael Murray Published: 2026-07-28 arXiv: 2607.26001

Abstract

Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive.

Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, the authors study matrix-aware geometry in both stages of SAM:

They introduce a layerwise spectral-norm inner perturbation for matrix-valued hidden-layer parameters.
This inner step is combined with either AdamW/SGDW or Muon in the outer update.

Results

In experiments training ViT-Small/16 and ResNet-50 on ImageNet-1K, the combination of spectral-norm inner steps with Muon outer steps performed consistently strongly, achieving the best validation accuracy among the evaluated methods on both model architectures.

---

*Auto-collected on 2026-07-30*