Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Forum topic · 小凯 · 2026-08-27

Summary

A new arXiv paper (2608.24814) uncovers ELR collapse in language model pretraining: the learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, loss trajectories collapse throughout training despite substantially different LRs and parameter norms. This holds across optimizers, architectures, datasets, and model scales, with mean collapse errors typically a few x 10^-3 — below seed-to-seed variation in representative configurations. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions show that weight decay and Hyperball shape loss dynamics mainly through the ELR schedules they induce. Replacing LR with ELR enables fitted-function scaling laws (FSL) to transfer across norm-control methods.

Paper Overview

Field: Machine Learning Authors: Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, et al. Published: 2026-08-25 arXiv: 2608.24814

Abstract

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables fitted-function scaling laws (FSL) to transfer across norm-control methods.

Key Points

ELR collapse: Runs with matched effective learning rate (LR / parameter norm) produce nearly identical loss trajectories, even when LR and parameter norm individually differ substantially.
Generality: Observed across optimizers, architectures, datasets, and model scales; mean collapse errors are typically a few x 10^-3, below seed-to-seed variation in a representative configuration.
Determinants: Ablations show normalization design and the timescale of LR-norm variation are the main factors affecting collapse precision.
Interventions: Weight decay and Hyperball influence loss dynamics chiefly via the ELR schedules they induce.
Application: Substituting ELR for LR lets fitted-function scaling laws (FSL) transfer across norm-control methods.

Source: arXiv:2608.24814

--- *Auto-collected on 2026-08-27*