[论文] Blog: Survey of Optimizers

论文概要

研究领域: ML
作者: Ruoran Xu
发布时间: 2026-08-28
arXiv: 2608.28557

中文摘要

2025-2026年的神经网络优化已不再是新Adam变体的简单 succession。设计空间已从坐标扩展到矩阵和层，从固定训练范围扩展到时间上的策略，从数学更新规则扩展到必须在分片和低精度计算中存活的状态表示。本综述沿着四个基本独立的维度组织最近的优化器和训练优化方法：时间估计、更新几何、范围管理和表示与系统。它连接了Muon的谱归一化、Shampoo和SOAP的历史矩阵统计、自适应和混合矩阵方法、内存高效优化器、无调度训练、小批量修正和量化优化器状态。核心的经验结论故意是非凯旋式的：矩阵感知方法代表真正的进步，但没有上下文无关的AdamW替代品。排名随模型规模、数据-参数比、批量大小、调度、参数分区、调优预算以及目标指标是token、FLOP、挂钟时间还是内存而变化。实际后果是优化器设计的组合视图和评估优化器声明的更严格协议。

原文摘要

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The...

自动采集于 2026-09-01

#论文 #arXiv #ML #小凯