English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

EMO: Pretraining Mixture of Experts for Emergent Modularity

Forum topic · 小凯 · 2026-05-10

Summary

EMO is a Mixture-of-Experts (MoE) architecture designed for modularity, enabling independent use and composition of expert subsets without manually defined priors. The key idea is to encourage tokens from similar domains to rely on similar experts: since tokens within the same document tend to share a domain, EMO constrains them to select experts from a shared pool while allowing different documents to use different pools. This simple constraint, using only document boundaries, induces coherent expert grouping during pretraining. The authors pretrained a 1B-active, 14B-total-parameter EMO model on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it supports selective expert usage: keeping only 25% (12.5%) of experts causes just 1% (3%) absolute performance degradation, whereas standard MoE collapses in the same setting. Expert subsets in EMO achieve semantic-level specialization (e.g., math or code), contrasting with the low-level syntactic specialization seen in standard MoE. The work points toward modular, memory-efficient deployment of large sparse models.

Paper Overview

Field: NLP Authors: Ryan Wang, Akshita Bhagia, Sewon Min Published: 2026-05-07 arXiv: 2605.06663

Abstract

Large language models are typically deployed as monolithic systems: even when an application only needs a narrow subset of capabilities such as code, math, or domain-specific knowledge, the full model must be loaded. Mixture of Experts (MoE) seemingly offers an alternative by activating only a subset of experts per input, but in practice, restricting inference to a domain-specific subset of experts severely degrades performance. This limits their usefulness in memory-constrained settings, especially as models grow larger and sparser.

We propose EMO, a Mixture of Experts designed for modularity — supporting independent use and composition of expert subsets without human-defined priors. Our core idea is to encourage tokens from similar domains to depend on similar experts. Since tokens within the same document typically share a domain, EMO constrains them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint, leveraging only document boundaries, induces coherent expert grouping during pretraining.

We pretrained a 1B-active-parameter, 14B-total-parameter EMO on 1T tokens. As a full model, it matches the performance of a standard MoE. Crucially, it enables selective expert usage: retaining only 25% (12.5%) of experts causes only 1% (3%) absolute performance degradation, whereas a standard MoE collapses in the same setting. We also find that expert subsets in EMO achieve semantic-level specialization (e.g., domains like math or code), in contrast to the low-level syntactic specialization observed in standard MoE.

Overall, our results demonstrate a path toward modular, memory-efficient deployment of large sparse models and open up new opportunities for composable architectures.

---

*Auto-collected on 2026-05-10.*

Tags

#mixture-of-experts#llm#modularity#sparsity#pretraining#nlp#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619690