Quantifying Hyperparameter Transfer and the Importance of Embedding Learning Rate in LLM Training

Forum topic · 小凯 · 2026-05-22

Summary

This paper by Dayal Singh Kalra and Maissam Barkeshli (arXiv:2505.15986, May 2025) studies hyperparameter transfer—the ability to extrapolate optimal training hyperparameters from small to large model scales, which is critical for training large language models. The authors first develop a framework quantifying hyperparameter transfer quality through three metrics: the quality of the scaling law fit, robustness to extrapolation errors, and the asymptotic loss penalty from the choice of parameterization. Through comprehensive ablation studies, they investigate why Maximal Update Parameterization (muP) provides superior learning rate transfer compared to standard parameterization (SP). Their key finding: when training with AdamW, muP's advantage over SP comes entirely from scaling the embedding layer learning rate by a factor of width. In SP, the embedding learning rate acts as a bottleneck causing training instability; increasing it to match muP significantly smooths training and improves hyperparameter transfer. They also find that weight decay improves scaling law fit quality but harms extrapolation robustness in fixed token-per-parameter settings. The work has practical implications for stabilizing LLM training and choosing parameterizations at scale.

Paper Overview

Field: Machine Learning
Authors: Dayal Singh Kalra, Maissam Barkeshli
Published: 2025-05-20
arXiv: 2505.15986

Abstract (Translated)

Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameterization, such as Maximal Update (muP), that renders optimal hyperparameters approximately scale invariant. In this paper, the authors first develop a framework to quantify hyperparameter transfer through three metrics: (1) the quality of the scaling law fit, (2) the robustness to extrapolation errors, and (3) the asymptotic loss penalty due to choice of parameterization. Next, they investigate through a comprehensive series of ablations why muP appears to offer high-quality learning rate transfer relative to standard parameterization (SP), since existing theory is insufficient to explain this.

Key Findings

When training with AdamW, muP's overwhelming advantage over SP comes entirely from maximizing the embedding layer learning rate.
In SP, the embedding layer learning rate acts as a bottleneck that induces training instability.
Scaling the embedding learning rate by a factor of width (matching muP) significantly smooths training while improving hyperparameter transfer.
Weight decay improves the quality of the scaling law fit, but in a fixed token-per-parameter setting it harms the robustness of extrapolation.

Original Abstract

Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameterization, such as Maximal Update (muP), that renders optimal hyperparameters approximately scale invariant. In this paper, we first develop a framework to quantify hyperparameter transfer through three metrics: (1) the quality of the scaling law fit, (2) the robustness to extrapolation errors, and (3) the asymptotic loss penalty due to choice of parameterization. Next, we investigate through a comprehensive series of ablations why muP appears to offer high-quality learning rate transfer relative to standard parameterization.

---

*Auto-collected on 2026-05-22*