Why Muon Outperforms Adam: A Curvature Perspective
A collaboration between Yale, NUS, and UMN offers the first curvature-based explanation of Muon's ~2x speedup over Adam in LLM pretraining: the advantage is not better gradient alignment, but a smaller second-order penalty.
Key Points
- Second-order Taylor decomposition: Per-step loss decrease splits into first-order gradient alignment benefit ⟨G, Z⟩ minus a curvature penalty ½⟨Z, H[Z]⟩. At matched validation loss (not matched steps), Adam and Muon show nearly identical first-order benefit, but Muon's curvature penalty is significantly smaller.
- NDS decomposition: The penalty factors into update norm ‖Z‖_F² and Normalized Directional Sharpness (NDS). The update norms are nearly equal (ratio ≈ 1.0), while Adam's NDS is 1.76x that of Muon. Muon's entire advantage comes from lower NDS, not larger steps.
- Why NDS is low: Muon's spectral normalization (O_t = U_t V_t^T from the SVD of the momentum-averaged gradient) discards gradient magnitude and keeps only direction. This regularizes the update direction, making it flatter on the Hessian and less likely to plunge into high-curvature directions and bounce back.
- Low Kronecker rank (r = 3–5 explains 80% of energy for W_K, W_V; 7–9 for W_Q, W_O)
- Simultaneous diagonalization efficiency ~0.85–0.89
- Curvature heterogeneity spanning 6 orders of magnitude (w_1/w_88 ≈ 2.59×10⁶)
- Gradient alignment: 80% of gradient energy in 30% of positive-curvature directions
- Wang, S., Zhang, F., Li, J., Bergemann, D., & Yang, Z., "Why Muon Outperforms Adam: A Curvature Perspective", arXiv:2606.04662, 2026
- Institutions: National University of Singapore, Yale University, University of Minnesota
- Validation models: 124M NanoGPT on FineWeb; 9M NanoGPT on Zipf-PCFG
Experiments
Zipf-PCFG synthetic data (9M NanoGPT)
| Imbalance s | Adam NDS | Muon NDS | Gap | |---|---|---|---| | 0 (balanced) | 1.63 | 1.00 | 0.63 | | 0.5 | 1.97 | 1.18 | 0.92 | | 1 (most imbalanced) | 2.38 | 1.25 | 1.13 |
The gap grows ~1.8x with data imbalance — explaining why Muon excels on real language data with Zipf-distributed token frequencies. Adam's NDS is amplified by imbalance; Muon is effectively immune.
Within-layer vs. cross-layer NDS
Muon's NDS shifts from cross-layer-dominated (14% within-layer early) to within-layer-dominated (44% late) — cross-layer coupling curvature drops fastest. Adam's ratio stays nearly flat. The within-layer gap concentrates in the first and deepest layers, where gradient propagation is least stable.
Theory
The LLM Hessian exhibits:
Theorem 5.5: On a quadratic model with heterogeneous curvature (groups w_H, w_L) and gradient alignment toward high-curvature directions, Muon's NDS is smaller than GD's for any finite T; if the curvature ratio ρ is large enough and the high-curvature group share is bounded, Muon also achieves lower loss. On a quadratic model, Muon vs. GD shows NDS ratio 0.09x and loss-decrease ratio 5.60x.
Conclusion
Muon's 2x speedup comes from being "blunter," not sharper:
1. Spectral normalization keeps only gradient direction; 2. Update energy is distributed uniformly across curvature directions; 3. NDS is lower, so the curvature penalty is smaller; 4. Per-step loss decrease is larger, so convergence is faster.
Data imbalance (Zipf-distributed rare tokens) and transformer layer structure amplify this advantage. The design lesson: rather than making updates more precise, make them smoother — on a high-curvature landscape, a blunt knife cuts deeper.