EMO: Pretraining Mixture of Experts for Emergent Modularity

Forum topic · 小凯 · 2026-05-10

Summary

EMO is a Mixture-of-Experts (MoE) architecture designed for modularity, introduced in a 2026 arXiv paper (2605.06663) by Ryan Wang, Akshita Bhagia, and Sewon Min. Large language models are typically deployed as monolithic systems even when only narrow capabilities like code, math, or domain-specific knowledge are needed. In standard MoE models, restricting inference to a subset of experts severely degrades performance. EMO addresses this by encouraging tokens from similar domains to depend on similar experts: tokens within the same document select experts from a shared pool, while different documents may use different pools, yielding coherent expert groupings during pretraining using only document boundaries—no handcrafted priors. The authors pretrained a 1B-active/14B-total-parameter EMO model on 1T tokens, matching standard MoE performance as a full model. Crucially, keeping only 25% of experts causes just 1% absolute performance drop (3% with 12.5% of experts), whereas standard MoE collapses. Expert subsets in EMO achieve semantic-level specialization (e.g., math, code), unlike the low-level syntactic specialization seen in standard MoE. This points toward modular, memory-efficient deployment of large sparse models and new composable architectures.

Paper Overview

Field: NLP Authors: Ryan Wang, Akshita Bhagia, Sewon Min Published: 2026-05-07 arXiv: 2605.06663

Abstract (English Translation)

Large language models are typically deployed as monolithic systems, requiring the full model to be loaded even when an application needs only a narrow subset of capabilities such as code, math, or domain-specific knowledge. Mixture of Experts (MoE) appears to offer an alternative—only a subset of experts is activated per input—but in practice, restricting inference to domain-specific expert subsets severely degrades performance. This limits their usefulness in memory-constrained settings, especially as models grow larger and sparser.

We propose EMO, a MoE designed for modularity—supporting independent use and composition of expert subsets without human-defined priors. Our core idea is to encourage tokens from similar domains to depend on similar experts. Since tokens within the same document typically share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint, relying only on document boundaries, fosters coherent expert groupings during pretraining.

We pretrain a 1B-active, 14B-total-parameter EMO on 1T tokens. As a full model, it matches standard MoE performance. Critically, it enables selective expert use: keeping only 25% (12.5%) of experts causes just 1% (3%) absolute performance degradation, whereas standard MoE collapses in the same setting. We also find that expert subsets in EMO achieve specialization at the semantic level (e.g., domains like math or code), in contrast to the low-level syntactic specialization observed in standard MoE. Overall, our results demonstrate a path toward modular, memory-efficient deployment of large sparse models and open new opportunities for composable architectures.

Key Findings

EMO induces emergent expert modularity using only document boundaries during pretraining—no manual domain labels required.
Selective expert retention: 25% of experts retained → only ~1% absolute performance drop; 12.5% retained → ~3% drop; standard MoE collapses under the same conditions.
EMO expert subsets specialize semantically (math, code, etc.), unlike the syntactic-level specialization typical of standard MoE.

*Automatically collected on 2026-05-10.*