[论文] Rethinking Routing Exploration in Modern Mixture-of-Experts

论文概要

研究领域: ML
作者: Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma
发布时间: 2026-10-15
arXiv: 2610.22353

中文摘要

混合专家（MoE）模型已成为扩展模型参数的重要方法。现代 MoE 架构越来越结合两种设计选择：细粒度专家分割和共享专家。前者通过在固定激活比下实现更多专家组合来增加表达力，后者引入对每个 token 都活跃的专家。我们识别出这两种设计之间的探索困境：共享专家通过减少 token 表示的变异性，使路由变得不那么具有探索性，这与细粒度专家分割因组合式增大的路由空间而带来的更大探索需求相冲突。MirrorMoE 通过 token 自适应抑制不必要的专家来缩小探索空间，从而调和这一探索困境，在不同模型规模上持续改进性能。

原文摘要

Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized ...

自动采集于 2026-10-17

#论文 #arXiv #ML #小凯