EMO: Pretraining Mixture of Experts for Emergent Modularity

Forum topic · 小凯 · 2026-05-09

Summary

EMO (arXiv:2505.03483) is a Mixture-of-Experts (MoE) architecture designed for modularity, enabling independent use and composition of expert subsets without human-defined priors. Standard MoEs degrade severely when inference is restricted to a domain-specific subset of experts, limiting their usefulness in memory-constrained settings. EMO's key idea is to encourage tokens from similar domains to rely on similar experts: since tokens within a document typically share a domain, EMO constrains them to select experts from a shared pool while allowing different pools across documents. This simple constraint, using only document boundaries, leads to coherent expert groupings emerging during pretraining. The authors pretrained a 1B-active-parameter, 14B-total-parameter EMO on 1T tokens. As a full model, it matches standard MoE performance. Notably, keeping only 25% (12.5%) of experts causes just 1% (3%) absolute performance drop, whereas standard MoE collapses entirely under the same conditions. Expert subsets in EMO also specialize at the semantic level (e.g., math or code domains), contrasting with the low-level syntactic specialization seen in standard MoEs. These results demonstrate a path toward modular, memory-efficient deployment of large sparse models.

Paper Overview

Field: NLP Authors: Ryan Wang, Akshita Bhagia, Sewon Min Published: 2026-05-06 arXiv: 2505.03483

Introduction

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser.

Method

The authors introduce EMO, an MoE designed for modularity — the independent use and composition of expert subsets — without requiring human-defined priors. The key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document typically share the same domain, EMO restricts them to selecting experts from a shared pool, while allowing different documents to use different pools. This simple constraint, relying only on document boundaries, enables coherent expert groupings to emerge during pretraining.

Results

The authors pretrained a 1B-active-parameter, 14B-total-parameter EMO on 1T tokens.
As a full model, EMO matches the performance of a standard MoE.
Selective expert usage: keeping only 25% (12.5%) of the experts results in only 1% (3%) absolute performance degradation, whereas a standard MoE completely collapses under the same setting.
Expert subsets in EMO specialize at the semantic level (e.g., math or code domains), in contrast to the low-level syntactic specialization observed in standard MoEs.

Conclusion

EMO demonstrates a path toward modular, memory-efficient deployment of large sparse models and opens new opportunities for composable architectures.

---

*Auto-collected on 2026-05-09*