Paper Overview
- Field: Machine Learning
- Author: Ruoran Xu
- Published: 2026-08-28
- arXiv: 2608.28557
- Spectral normalization in Muon
- Historical matrix statistics in Shampoo and SOAP
- Adaptive and hybrid matrix methods
- Memory-efficient optimizers
- Schedule-free training
- Small-batch corrections
- Quantized optimizer states
- Paper: https://arxiv.org/abs/2608.28557
Summary
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation.
This survey organizes recent optimizers and training optimization methods along four largely independent axes:
1. Temporal estimation — how gradients and statistics are estimated over time 2. Update geometry — the shape and structure of parameter updates (coordinate-wise vs. matrix-aware) 3. Horizon management — fixed training schedules vs. schedule-free policies 4. Representation and systems — optimizer state design under sharding and low-precision constraints
Topics Covered
Key Takeaway
The core empirical conclusion is deliberately non-triumphal: matrix-aware methods represent real progress, but there is no context-free replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partitioning, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory.
The practical consequences are a compositional view of optimizer design and stricter protocols for evaluating optimizer claims.
Links
*Auto-collected on 2026-09-01.*