Paper Overview
Research Area: NLP Authors: Songhao Wu, Ang Lv, Ruobing Xie, Yankai Lin Published: 2026-06-10 arXiv: 2606.12397
Abstract (English)
Router is the cornerstone component of Mixture-of-Experts (MoE) models. Serving as expert proxies, the rows of the router matrix compute their similarity to MoE inputs to determine which subset of experts is activated. Ideally, each router row should encode its associated expert matrix into a representative vector, so that its dot product with a token better reflects token-expert affinity. However, no design principle enforces this condensation. This paper proposes aligning each router row with the principal singular direction of its associated expert, since this direction provides the most expressive mathematical description of a matrix.
Based on this principle, the authors propose redesigning the router with Manifold Power Iteration (MPI). Specifically, MPI introduces a Power-then-Retract paradigm: a power iteration step is performed on the router weights, followed by a retraction step that imposes norm constraints, ensuring efficiency and stability.
- Theoretical result: MPI-driven router rows provably converge to the principal singular direction of the associated expert matrix.
- Empirical result: Pretraining MoE models at scales from 1B to 11B parameters confirms that this alignment leads to more effective MoE models.
- arXiv: https://arxiv.org/abs/2606.12397