Summary
This survey paper (arXiv:2608.28557, Ruoran Xu, published 2026-08-28) argues that neural-network optimization in 2025-2026 can no longer be described as a simple succession of Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. The paper organizes recent optimizers along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects Muon's spectral normalization, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical finding is deliberately non-triumphalist: matrix-aware methods represent genuine progress, but there is no context-free replacement for AdamW. Rankings vary with model scale, data-to-parameter ratio, batch size, schedule, parameter partitioning, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. Practical implications include a combinatorial view of optimizer design and stricter protocols for evaluating optimizer claims.
Overview
Field: Machine Learning
Author: Ruoran Xu
Published: 2026-08-28
arXiv: 2608.28557
Summary
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation.
This survey organizes recent optimizers and training optimization methods along four largely independent axes:
1. Temporal estimation — how statistics are accumulated and refreshed over time
2. Update geometry — coordinate-wise vs. matrix- and layer-aware updates
3. Horizon management — fixed training horizons vs. policies over time (e.g., schedule-free training)
4. Representation and systems — optimizer state that survives sharding and low-precision computation
Methods Covered
- Spectral normalization in Muon
- Historical matrix statistics in Shampoo and SOAP
- Adaptive and hybrid matrix methods
- Memory-efficient optimizers
- Schedule-free training
- Small-batch corrections
- Quantized optimizer states
Key Findings
The core empirical conclusion is deliberately non-triumphalist: matrix-aware methods represent genuine progress, but there is no context-free replacement for AdamW. Rankings vary with:
- Model scale and data-to-parameter ratio
- Batch size and schedule
- Parameter partitioning and tuning budget
- Target metric: tokens, FLOPs, wall-clock time, or memory
The practical consequences are a
combinatorial view of optimizer design and
stricter protocols for evaluating optimizer claims.
---
*Auto-collected on 2026-09-01.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634339