MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control — New Muon Variant Explained

Forum topic · 小凯 · 2026-10-28

Summary

MuonGS is a new optimizer for language model pretraining introduced in arXiv paper 2610.37647 by Athanasios Glentis, Dawei Li, and Mingyi Hong. The work analyzes weight-norm control in the Muon optimizer family through the accumulated effective learning rate C_T = Σ η_t/||W_t||_F. The authors show vanilla Muon accumulates only logarithmically, potentially limiting exploration, while weight-decay variants like MuonW and MuonH guarantee at least linear accumulation. Their key insight is that sufficiently slow norm growth already guarantees faster-than-logarithmic accumulation, enabling less restrictive optimizer designs. MuonGS applies Gram–Schmidt projection to remove the radial component of each Muon update without fixing the weight norm, proving the weight norm grows at most proportionally to √t and guaranteeing C_T = Ω(√T), plus an O(T^{-1/2}) stationarity rate for smooth scale-invariant objectives. Experiments on 160M and 400M dense models and a 135M-active/1B-total-parameter MoE model show MuonGS achieves lower validation perplexity than MuonH at long training horizons, with an estimated token-equivalent speedup of up to ~15% for dense models at matched validation loss.

Paper Overview

Field: ML / Optimization Authors: Athanasios Glentis, Dawei Li, Mingyi Hong arXiv: 2610.37647

Why Weight-Norm Control Matters

Weight decay and explicit weight-norm constraints are known to improve optimizer performance in language model (LLM) pretraining, but a unified understanding of *why* such control helps and *how much* is needed has remained incomplete. This paper provides that analysis for the Muon optimizer family.

Key Contributions

Accumulated effective learning rate: The authors analyze norm control over the entire training trajectory via

C_T = \sum_{t<T} \frac{\eta_t}{\|W_t\|_F}

where $\eta_t$ is the nominal learning rate and $\eta_t/\|W_t\|_F$ is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds the relative weight displacement.

Diagnosis of vanilla Muon: Vanilla Muon can accumulate only *logarithmically*, which may limit exploration and helps explain its slower empirical progress. In contrast, MuonW and MuonH guarantee at least linear accumulation under the stated conditions.
Relaxed control is enough: Sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening the door to less restrictive optimizer designs.

The MuonGS Optimizer

Based on this insight, MuonGS uses Gram–Schmidt (GS) projection to remove the radial component of each Muon update *without fixing the weight norm*. With bounded update directions and a constant nominal learning rate:

The weight norm grows at most proportionally to $\sqrt{t}$ after $$t$$ steps.
MuonGS guarantees $C_T = \Omega(\sqrt{T})$ , with accumulation sustained even in later training.
For smooth scale-invariant objectives, an $O(T^{-1/2})$ stationarity guarantee is established.

Experimental Results

Pretraining experiments covered:

160M- and 400M-parameter dense models
A MoE model with 135M active / 1B total parameters

Findings: MuonGS achieves lower validation perplexity than MuonH at long training horizons. For dense models, it achieves an estimated token-equivalent speedup of up to about 15% at matched validation loss versus MuonH.

Source

Original abstract: arXiv:2610.37647

--- *Automatically collected on 2026-10-29*