Streaming Matrix Computations in Training: Streaming Newton-Schulz Orthogonal Projection and mclip Spectral Norm Clipping

Forum topic · ✨步子哥 · 2026-04-20

Summary

This post explains how streaming (incremental) matrix computations extend the Muon optimizer by amortizing expensive matrix decompositions across training steps. It covers two core techniques: (1) streaming Newton-Schulz iteration, which keeps parameter matrices near the Stiefel manifold (orthogonality constraint W^T W = I) by maintaining an auxiliary matrix X_t that is softly blended with the current weights and updated with a single Newton-Schulz step per iteration, achieving approximate orthogonal projection at roughly O(mn) cost instead of O(mn^2) full SVD; and (2) mclip, a streaming spectral norm clipping method that uses Muon's maintained left/right singular vector estimates to subtract a rank-1 correction whenever the largest singular value exceeds 1, enforcing Lipschitz constraints for adversarial robustness and GAN stability. Experiments cited show orthogonality error stable at 1e-3 to 1e-2, spectral norms held below 1, and training speedups of 2-5x with negligible accuracy loss. The article also touches on streaming matrix square roots, inverse estimation, and Oja's rule, framing streaming as a general incremental-computation paradigm well-suited to large-scale model training.

Overview

This article (from zhichai.net, based on Jianlin Su's Muon optimizer series at kexue.fm/archives/11719) introduces streaming matrix computations as a way to replace expensive one-shot decompositions (SVD, polar decomposition) with cheap per-step updates embedded in the training loop.

Key idea

Parameter matrices change very little between training steps (learning rates ~1e-3 to 1e-5), so quantities like the orthogonal projection or the largest singular value vary slowly and can be tracked incrementally rather than recomputed from scratch.
Maintain a state S_t and update it cheaply each step: S_t ← Update(S_{t-1}, W_t), keeping S_t ≈ f(W_t).

1. Streaming Newton-Schulz for orthogonal projection (Stiefel manifold)

The Stiefel manifold St(m,n) = {W : W^T W = I_n} preserves signal norms (||Wx||₂ = ||x||₂), mitigating gradient vanishing/explosion and smoothing the optimization landscape.
Standard approach: polar decomposition via full SVD, O(mn²) per step — too expensive.
Streaming version:
Only two extra matrix multiplications per step. Orthogonality error ||W^T W − I||_F stays around 1e-3–1e-2 in experiments; small errors self-correct over subsequent steps. Reported 2–5x training speedup with negligible performance loss.

2. mclip: streaming spectral norm clipping

The spectral norm ||W||₂ (largest singular value) is the Lipschitz constant of a linear layer; clipping it to ≤ 1 aids adversarial robustness, generalization, and GAN stability.
Traditional clipping W ← W / max(1, ||W||₂) requires SVD or many power iterations.
mclip reuses Muon's maintained left/right singular vector estimates u_t, v_t:
σ₁ ≈ u_t^T W v_t
W ← W − max(0, σ₁ − 1) · u_t v_t^T
Cost: one matrix-vector product, one outer product, one subtraction — O(mn). It clips only the largest singular value each step; because updates are small, clipping one direction per step converges globally to all σᵢ ≤ 1 (at most n steps for a fixed W), analogous to projected gradient descent.

Cost comparison

| Method | Complexity | Constant factor | |---|---|---| | Full SVD projection | O(mn²) | large (~10–20n) | | Multi-step Newton-Schulz | O(kmn²) | moderate (~k) | | Spectral Normalization | O(mn + n²) | small | | Streaming Newton-Schulz | O(mn) | tiny | | mclip | O(mn) | tiny |

Streaming methods scale linearly with parameter size, offering order-of-magnitude speedups for large models.

Other streaming tricks

Matrix square root: Y_{k+1} = ½ Y_k (3I − Y_k²), streamed as one step per iteration.
Approximate inverse: Z_{k+1} = 2Z_k − Z_k A Z_k.
Streaming PCA via Oja's rule: v_{t+1} = v_t + η_t (x_t x_t^T v_t − (v_t^T x_t x_t^T v_t) v_t) — spiritually related to Muon's streaming power iteration.

Conclusion

Streaming Newton-Schulz (orthogonality) and mclip (Lipschitz constraints) complete the Muon ecosystem: complex decompositions become gentle per-step refinements, yielding major efficiency gains with minimal accuracy cost. Open directions include rigorous streaming theory, integration with other efficient techniques, and validation at trillion-parameter scale.

References

1. Jianlin Su. Muon optimizer via streaming power iteration, Part 5: Extensions. Scientific Spaces, kexue.fm/archives/11719. 2. Jianlin Su. Earlier Muon optimizer series posts (streaming power iteration). 3. Polar decomposition and Newton-Schulz iteration literature (Stiefel manifold applications). 4. Spectral norm and Lipschitz constraints in GANs and adversarial training. 5. Unified streaming computation frameworks and Oja's rule extensions.