Redesigning Mixture-of-Experts Routers with Manifold Power Iteration

Forum topic · 小凯 · 2026-06-12

Summary

This post summarizes an arXiv paper (2606.12397) proposing a redesign of Mixture-of-Experts (MoE) routers based on Manifold Power Iteration (MPI). The authors — Songhao Wu, Ang Lv, Ruobing Xie, and Yankai Lin — observe that router rows, which act as proxies for experts by computing similarity with MoE inputs, lack a principled design ensuring they encode the expert matrices they represent. The paper proposes aligning each router row with the principal singular direction of its associated expert, since this direction offers the most expressive mathematical description of a matrix. The MPI method uses a Power-then-Retract paradigm: a power iteration step is applied to router weights, followed by retraction to enforce norm constraints, ensuring efficiency and stability. Theoretically, the authors prove that MPI-driven router rows converge to the principal singular direction of the associated expert. Empirically, they pretrain MoE models at scales from 1B to 11B parameters and confirm that this alignment yields more effective MoE models. The work spans NLP and MoE architecture design.

Paper Overview

Field: NLP Authors: Songhao Wu, Ang Lv, Ruobing Xie, Yankai Lin arXiv: 2606.12397

Summary

Routers are the cornerstone component of Mixture-of-Experts (MoE) models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row should encode the expert matrix into this representative vector, so that its dot product with a token better reflects token-expert affinity. However, no design principles exist to enforce this condensation.

This paper proposes aligning each router row with the principal singular direction of the associated expert, as this direction provides the most expressive mathematical description of a matrix. Based on this principle, the authors redesign routers with Manifold Power Iteration (MPI). Specifically, MPI introduces a *Power-then-Retract* paradigm: a power iteration step is performed on the router weights, followed by retraction to impose norm constraints, ensuring efficiency and stability.

Theoretically, the authors prove that MPI-driven router rows converge to the principal singular direction of the associated expert. Empirically, MoE models are pretrained at scales from 1B to 11B parameters, confirming that this alignment promotes more effective MoE models.

Key Contributions

A design principle: router rows should align with the principal singular direction of their associated experts
Manifold Power Iteration (MPI) with a Power-then-Retract paradigm for efficient, stable optimization
Convergence guarantee: router rows provably converge to the principal singular direction
Empirical validation via pretraining MoE models from 1B to 11B parameters

---

*Source: arXiv:2606.12397, posted on zhichai.net.*

Redesigning Mixture-of-Experts Routers with Manifold Power Iteration

Summary

Paper Overview

Summary

Key Contributions

Tags