English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

The SGD-Adam Gap in LLM Pretraining Isn't About Adaptivity—It's About Learning Rate Size

Forum topic · 小凯 · 2026-05-19

Summary

A forum post discusses why SGD underperforms Adam in LLM pretraining, challenging the standard explanation that Adam's per-parameter adaptive learning rates are essential. Citing work by Glentis, Li, Yau, and Hong, the author argues the gap stems from effective learning rate magnitude. LLM pretraining features small gradient norms and large weight-to-gradient ratios, requiring large effective step sizes. Adam naturally amplifies its update via gradient normalization, while SGD cannot. The limiting factor is the uneven gradient distribution at the output layer: rare tokens occasionally produce huge gradient spikes that cap the maximum learning rate SGD can tolerate. With simple gradient clipping, SGD pretraining a 1B-parameter LLaMA model closed a validation loss gap from over 50% versus Adam down to just 3.5%. Open questions remain: the cause of the residual 3.5% gap, automatic clipping threshold selection, and whether findings hold at smaller batch sizes beyond the 1M-token batches used.

Why does SGD perform far worse than Adam when pretraining large language models? The widely accepted explanation is Adam's adaptive learning rate mechanism—adjusting the step size per parameter—which offers an essential advantage in the high-dimensional, sparse-gradient setting of LLMs. Glentis, Li, Yau, and Hong revisited this assumption and found the answer may be simpler: the gap is largely about effective learning rate size, not adaptivity itself.

Key points

  • LLM pretraining has small gradient norms and large weight-to-gradient ratios—especially with large batch sizes. This means a large effective learning rate is needed for weight updates to matter materially.
  • Adam's updates are not bounded by gradient norms: its gradient-to-gradient-norm normalization mechanism naturally amplifies the effective step size. SGD lacks this mechanism.
  • The bottleneck is uneven gradient distribution at the output layer. Gradient magnitudes vary enormously across token classes—common words ("the", "a") have small gradients, while rare words occasionally produce huge gradient spikes in certain batches. These spikes cap the maximum learning rate SGD can withstand; a slightly larger learning rate and a single spike can blow up the whole model.
  • Gradient clipping closes most of the gap. With simple gradient clipping, pretraining a 1B-parameter LLaMA model on a large dataset, SGD's validation loss gap versus Adam shrank from over 50% to just 3.5%.
  • Clipping allows SGD to run stably at large learning rates—which is exactly the advantage Adam has always implicitly enjoyed but that went unnoticed.
  • Open questions

  • What causes the remaining 3.5% gap—other benefits of adaptive learning rates, such as handling differences in parameter scales?
  • How should the clipping threshold be determined automatically?
  • The paper uses very large batches of 1M tokens—do the findings still hold under the more common smaller-batch settings?
---

References

1. Glentis, A., Li, D., Yau, C., & Hong, M. (2026). *Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates*. arXiv:2605.17787 [cs.LG]. 2. Kingma, D. P., & Ba, J. (2015). *Adam: A Method for Stochastic Optimization*. ICLR. 3. Zhang, J., et al. (2020). *Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity*. ICLR.

Tags

#sgd#adam-optimizer#llm-pretraining#gradient-clipping#optimization#learning-rate#deep-learning#machine-learning-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620382