Paper Overview
Field: ML Authors: Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma Published: 2026-10-15 arXiv: 2610.22353
Chinese Summary (translated)
Mixture-of-Experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio; the latter introduces experts that are active for every token. The authors identify an exploration dilemma between these designs: shared experts make routing less exploratory by reducing the variability of token representations, which conflicts with the greater exploration need induced by fine-grained expert segmentation due to its combinatorially enlarged routing space.
Original Abstract (excerpt)
> Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized ...
Proposed Method
MirrorMoE narrows the exploration space by token-adaptively suppressing unnecessary experts, reconciling the exploration dilemma. The authors report consistent performance improvements across different model scales.
---
*Auto-collected on 2026-10-17*