Survey of Optimizers: Four Axes of Neural Network Optimization (arXiv 2608.28557)

Forum topic · 小凯 · 2026-09-01

Summary

This survey paper (arXiv:2608.28557) by Ruoran Xu, posted on zhichai.net, reviews the state of neural network optimizers in 2025-2026. It argues that the field has moved beyond successive Adam variants: the design space now spans coordinates to matrices and layers, fixed training horizons to time-varying policies, and mathematical update rules to state representations that must survive sharding and low-precision computation. The survey organizes recent optimizers along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects Muon's spectral normalization, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The core empirical conclusion is deliberately non-triumphalist: matrix-aware methods represent genuine progress, but there is no context-free AdamW replacement. Rankings vary with model scale, data-to-parameter ratio, batch size, schedules, parameter partitioning, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. Practical takeaways include a combinatorial view of optimizer design and stricter protocols for evaluating optimizer claims.

Survey of Optimizers

Field: Machine Learning Author: Ruoran Xu Published: 2026-08-28 arXiv: 2608.28557

Abstract

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation.

This survey organizes recent optimizers and training optimization methods along four largely independent axes:

1. Temporal estimation 2. Update geometry 3. Horizon management 4. Representation and systems

It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states.

Key Findings

Matrix-aware methods represent genuine progress, but there is no context-free replacement for AdamW.
Optimizer rankings vary with model scale, data-to-parameter ratio, batch size, schedules, parameter partitioning, tuning budget, and the target metric (tokens, FLOPs, wall-clock time, or memory).
The practical consequence is a combinatorial view of optimizer design and stricter protocols for evaluating optimizer claims.

--- *Auto-collected on 2026-09-01.*