HyperP: Why Constraining Weights to a Hypersphere Makes Large Model Training More Stable

> Imagine climbing a mountain. Traditional optimization is like walking on a boundless plain — you can move fast, but you can also drift endlessly off course. Hypersphere optimization is like walking on the surface of a giant sphere: you never fall in or fly out, and every step stays within a naturally bounded region. That is the core idea of HyperP.

Key points

• HyperP constrains weight matrices to a fixed-radius Frobenius sphere: after each update, weights are re-projected via W ← C · (W - η·G) / ||W - η·G||_F.
• On the sphere, output magnitude is structurally bounded: ||Wx||₂ ≤ C · ||x||₂, so logit explosions and activation outliers are controlled without z-loss.
• Weight decay becomes a first-order no-op on the Frobenius sphere: tangential projection gives Π_T(W) = 0, so the decay term vanishes — one fewer hyperparameter to tune.
• Width scaling is handled automatically by the norm constraint; depth still requires Depth-μP-style scaling (η ∝ 1/√d), contrary to MuonH's claim of being "inherently depth-transferable."
• Data scaling yields η* = 24.27 · T^(-0.32), exactly matching prior AdamW findings — suggesting a universal exponent for gradient-based optimization.
• SqrtGate for MoE: replacing softmax gate weights g_i with √g_i keeps routed output RMS independent of top-k (||y_route'||_rms ≈ r), improving loss at k = 2/8/32 and reducing router Z-value peaks by 5×.
• Transferable stability: six instability metrics (attention/router Z-values, output RMS, outlier percentages) all stay bounded or *decrease* with depth — stability validated at small scale transfers (and improves) at large scale.
• Compute efficiency: at 6×10²¹ FLOPs, MuonH+HyperP achieves 1.58× compute-efficiency gain over Muon (monotonically increasing with scale: 0.99× at d=8 up to 1.58× at d=24), and 3.38× over a dense baseline for MoE.
• Architecture ablations show Gated Attention + QK-Norm is best across scales (with diminishing marginal returns), while SqrtGate and Shared Expert provide orthogonal MoE gains (best combined loss: 2.3154).

The problem: instability at scale

LLM training has scaled parameters 1000× (1B → 1T), data 100×, and compute 1000× (10²¹ → 10²⁴ FLOPs) — but scaling brings instability: logit explosions, activation outliers (>5σ), loss spikes, and dependence on z-loss regularization. Existing methods (AdamW + μP/μP++) require tuning learning rate, weight decay, batch size, and z-loss coefficients, whose optima shift with scale. HyperP reduces this to a one-dimensional search over learning rate alone.

Why the hypersphere works

The update decomposes into radial and tangential components; tangential projection preserves only the tangent part: W⁺ - W = Π_T(Δ) + O(||Δ||²). Since W is normal to the sphere, Π_T(W) = 0, so weight decay (Δ = -η·G - η·λ·W) is projected away entirely. Geometrically: *you can slide on the sphere, but you can't push it further out or pull it in.*

Experimental setup

• Architecture: Transformer-Next (QK-Norm + Gated Attention), SlimPajama dataset
• Scale: 208M–3.8B active params (dense), 913M–13.3B total (MoE), 50 tokens per parameter
• Compared: Muon (μP++, weight decay ∝ 1/w), MuonH (1/√d_in init, no weight decay, Frobenius constraint), MuonH+HyperP (full scaling rules)

Practical recipe

1. Set weight decay λ = 0; remove z-loss. 2. Use MuonH for matrix weights, AdamH for vectors/embeddings. 3. Apply HyperP scaling: width automatic, depth η ∝ 1/√d, data η ∝ T^(-0.32). 4. For MoE, replace standard softmax gating with SqrtGate.

Limitations and open questions

Theory mainly covers single layers; deep-network cross-layer Jacobians are highly anisotropic. The Chinchilla compute-optimality assumption and the empirical 0.32 exponent lack universal theoretical grounding. Applicability to linear recurrent models (e.g., Mamba), RL, multimodal training, and 100B+ scale remains to be validated.

References

• Ren, L., Liu, Y., Shen, Y., & Chen, W. (2026). *Rethinking Language Model Scaling under Transferable Hypersphere Optimization*. arXiv:2603.28743.
• Yang et al. (2022). *Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer* (μP).
• Jordan et al. (2024). *Muon: An optimizer for hidden layers in neural networks*.
• Hoffmann et al. (2022). *Training Compute-Optimal Large Language Models* (Chinchilla).
• Zoph et al. (2022). *ST-MoE: Designing Stable and Transferable Sparse Expert Models* (z-loss).
• Code: https://github.com/microsoft/ArchScale

The takeaway: sometimes constraints are not limitations but liberation. Restricting weights to a sphere makes optimization simpler, more stable, and more predictable — a single small-scale tuning run transfers across scales.