Why does SGD perform far worse than Adam when pretraining large language models? The widely accepted explanation is Adam's adaptive learning rate mechanism—adjusting the step size per parameter—which offers an essential advantage in the high-dimensional, sparse-gradient setting of LLMs. Glentis, Li, Yau, and Hong revisited this assumption and found the answer may be simpler: the gap is largely about effective learning rate size, not adaptivity itself.
Key points
- LLM pretraining has small gradient norms and large weight-to-gradient ratios—especially with large batch sizes. This means a large effective learning rate is needed for weight updates to matter materially.
- Adam's updates are not bounded by gradient norms: its gradient-to-gradient-norm normalization mechanism naturally amplifies the effective step size. SGD lacks this mechanism.
- The bottleneck is uneven gradient distribution at the output layer. Gradient magnitudes vary enormously across token classes—common words ("the", "a") have small gradients, while rare words occasionally produce huge gradient spikes in certain batches. These spikes cap the maximum learning rate SGD can withstand; a slightly larger learning rate and a single spike can blow up the whole model.
- Gradient clipping closes most of the gap. With simple gradient clipping, pretraining a 1B-parameter LLaMA model on a large dataset, SGD's validation loss gap versus Adam shrank from over 50% to just 3.5%.
- Clipping allows SGD to run stably at large learning rates—which is exactly the advantage Adam has always implicitly enjoyed but that went unnoticed.
- What causes the remaining 3.5% gap—other benefits of adaptive learning rates, such as handling differences in parameter scales?
- How should the clipping threshold be determined automatically?
- The paper uses very large batches of 1M tokens—do the findings still hold under the more common smaller-batch settings?
Open questions
References
1. Glentis, A., Li, D., Yau, C., & Hong, M. (2026). *Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates*. arXiv:2605.17787 [cs.LG]. 2. Kingma, D. P., & Ba, J. (2015). *Adam: A Method for Stochastic Optimization*. ICLR. 3. Zhang, J., et al. (2020). *Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity*. ICLR.