Paper Overview
Field: Machine Learning Authors: Catherine Li, Jingchu Gai, Andrej Risteski, Jacob Mitchell Springer, Aditi Raghunathan Posted: 2026-11-02 arXiv: 2611.03633
Summary
Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning. This paper studies the downstream behavior of Muon and AdamW and uncovers a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token-parameter ratios.
To explain this reversal, the authors view fine-tuning as a perturbation of model outputs and introduce the margin-to-spread ratio, which measures how far the correct logit lies above the wrong ones relative to how widely the wrong logits are spread. In language models, Muon yields a larger ratio at small data budgets, while AdamW yields a larger one at large budgets — consistent with the fine-tuning trends.
In linear associative memory models, the authors prove that after one update, gradient descent's margin-to-spread ratio surpasses Muon's as the data budget grows; simulations show this trend persists after long training.
Key Findings
- Muon accelerates pretraining loss reduction vs. AdamW, but AdamW wins after fine-tuning once the pretraining data budget exceeds a threshold.
- The crossover token-parameter ratio shrinks as model size grows.
- The margin-to-spread ratio explains the reversal empirically (language models) and theoretically (linear associative memory).
Original Abstract (excerpt)
> Muon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning. In this paper, we study the downstream behavior of Muon and AdamW and uncover a surprising trend: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning. Moreover, as model size increases, AdamW overtakes Muon at progressively smaller pretraining token-parameter ratios. To explain this reversal, we view fine-tuning as a perturbation of the outputs and introduce the margin-to-spread ratio...
--- *Auto-collected on 2026-11-04*