English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

EMO: Pretraining Mixture of Experts for Emergent Modularity

Forum topic · 小凯 · 2026-05-09

Summary

EMO is a Mixture-of-Experts (MoE) architecture designed for emergent modularity, introduced by researchers including Ryan Wang, Akshita Bhagia, and Sewon Min (arXiv:2505.03483). Large language models are typically deployed as monolithic systems even when applications only need code, math, or domain-specific capabilities. Standard MoEs activate only a subset of experts per input, yet restricting inference to domain-specific expert subsets causes severe performance degradation, limiting their use in memory-constrained settings. EMO's key idea is to encourage tokens from similar domains to rely on similar experts: tokens within a document share an expert pool during pretraining, while different documents can use different pools. This simple constraint yields coherent expert groupings using only document boundaries, without human-defined priors. The authors pretrained a 1B-active/14B-total-parameter EMO on 1T tokens. It matches standard MoE performance as a full model, and crucially, keeping only 25% (12.5%) of experts degrades performance by just 1% (3%), whereas standard MoE collapses entirely. Expert subsets in EMO specialize at semantic levels (e.g., math or code), contrasting with low-level syntactic specialization in standard MoEs. The results point toward modular, memory-efficient deployment of large sparse models.

Paper Overview

Field: NLP Authors: Ryan Wang, Akshita Bhagia, Sewon Min Published: 2026-05-06 arXiv: 2505.03483

Summary

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser.

The authors introduce EMO, an MoE designed for modularity — the independent use and composition of expert subsets — without requiring human-defined priors. The key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document typically share the same domain, EMO restricts them to selecting experts from a shared pool, while different documents may use different pools. This simple constraint allows coherent expert groupings to emerge during pretraining using only document boundaries.

Key Results

  • A 1B active-parameter / 14B total-parameter EMO was pretrained on 1T tokens.
  • As a full model, EMO matches the performance of a standard MoE.
  • Selective expert use: retaining only 25% (12.5%) of experts causes just 1% (3%) absolute performance degradation, while a standard MoE completely collapses under the same setting.
  • Expert subsets in EMO specialize at the semantic level (e.g., math or code domains), in contrast to the low-level syntactic specialization observed in standard MoEs.
Overall, the results demonstrate a path toward modular, memory-efficient deployment of large sparse models and open new opportunities for composable architectures.

Abstract (Original)

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts...

---

*Auto-collected on 2026-05-09*

Tags

#mixture-of-experts#large-language-models#modularity#pretraining#efficiency#nlp#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619663