EMO: Emergent Modularity in Pretrained MoE — A Paradigm Shift for Sparse Experts

Forum topic · 小凯 · 2026-05-08

Summary

Researchers from UC Berkeley and the Allen Institute for AI propose EMO (Emergent Modularity via pretraining MoE), a method that induces semantic-level expert modularity in Mixture-of-Experts models during pretraining. EMO uses document boundaries as free weak supervision: all tokens within a document must route through a shared expert pool of randomly sampled size, combined with global load balancing. Training a 1B-active/14B-total parameter model with 128 experts on 1T tokens (OLMoE corpus), EMO matches standard MoE full-model performance on MMLU (42.8 vs 42.4) while drastically improving selective expert deployment: keeping only 25% of experts yields just a ~1% MMLU drop versus ~10% for standard MoE, and expert selection becomes robust to pruning methods. Token clustering and domain-similarity analyses show expert specialization shifts from syntactic (prepositions, articles) to semantic (reviews, medical, code). The work enables memory-constrained deployment, structured interpretability, content governance via expert disabling, and modular continual learning. Code, models, and visualizations are open-sourced.

EMO: Emergent Modularity in Pretrained MoE — A Paradigm Shift for Mixture-of-Experts Architecture

> TL;DR: UC Berkeley and the Allen Institute for AI propose EMO, achieving emergent modularity in MoE pretraining via a document-level expert pool constraint. 1B/14B parameter models trained on 1T tokens retain performance within ~1% when only 25% of experts are kept (standard MoE drops ~10%), and expert specialization shifts from syntactic to semantic levels. Code, models, and visualizations are open-sourced.

1. Problem: The Sparsity Paradox of MoE

Since Shazeer et al. (2017), Mixture-of-Experts (MoE) has become a core pillar of LLM scaling. Frontier systems like DeepSeek-V3, Qwen3, and Grok-1 rely on sparse activation to match dense-model expressiveness at far lower compute. Formally, a token's hidden state $$x_t$$ passes through a router:

p_t = \text{softmax}(r(x_t)) \in \mathbb{R}^{n_r}, \quad R_t = \text{TopK}(p_t, k)

\text{FFN}_{\text{out}}(x_t) = \sum_{i \in R_t} (p_t)_i \cdot E_i(x_t) + \sum_{j=1}^{n_s} E_j^{(s)}(x_t)

where the second term is a shared expert (always active, $$n_s$$ usually 1).

However, there is a deployment-level paradox:

| Dimension | Theoretical promise | Actual deployment | |:-----|:---------|:---------| | Compute | Only $$k$$ experts activated | ✓ Achieved | | Memory | Only active experts loaded | ✗ All $$n_r + n_s$$ experts must be loaded | | Specialization | Task/domain division | ✗ Syntactic division |

Prior work shows standard MoE experts specialize at the surface syntactic level — prepositions, punctuation, articles, subword fragments. Consequently, even a task needing only one domain (e.g., math reasoning) activates most experts, because every token seeks its own "grammar expert." MoE is sparse only in forward compute, remaining dense in memory — and VRAM is now the primary constraint as models scale toward trillions of parameters.

2. Method: Document Boundaries as Weak Supervision

EMO's core idea: constrain all tokens within a document to share one expert pool, using document boundaries that pretraining corpora already contain.

1. Compute the document-level average routing distribution: $\bar{p} = \frac{1}{T} \sum_{t=1}^{T} \text{softmax}(r(x_t))$ 2. Select a Top- $$d$$ document expert pool: $D = \text{TopK}(\bar{p}, d)$ 3. Restrict each token's experts to $$D$$ with conditional renormalization:

\hat{p}_t(i) = \frac{(p_t)_i}{\sum_{j \in D} (p_t)_j} \text{ if } i \in D, \text{ else } 0

Renormalization is necessary to keep routing weights summing to 1 and preserve the weighted sum invariance of expert combinations.

Dynamic pool size: fixing $$d$$ would make the model rigid at inference. Instead $d \sim \mathcal{U}\{k, k+1, \ldots, n_r\}$ , sampled independently per document, exposing the model to all possible subset sizes during pretraining.

Global load balancing: standard micro-batch load balancing conflicts with document-level pooling (it scatters a document's tokens across experts). EMO aggregates routing statistics across data-parallel groups: $\mathcal{L}_{\text{LB}} = n_r \sum_{i=1}^{n_r} \bar{f}_i \cdot \bar{P}_i$ — global balancing governs cross-document utilization while the shared-pool constraint governs within-document consistency. The two are complementary, not conflicting.

3. Experimental Setup

| Config | Value | |:-------|:-----| | Active / total params | 1B / 14B | | Routed experts $$n_r$$ | 127 (+1 shared) | | Experts per token $$k$$ | 8 | | Pretraining data | 1T tokens (OLMoE corpus) | | Annealing | 50B tokens (linear) | | Baseline | Same-architecture standard MoE, same data/scale |

Evaluation covers full-model zero-shot (MC9, Gen5, MMLU, MMLU-Pro, GSM8K) and Selective Expert Use, using Router-based selection, Easy-EP (SOTA expert pruning), and Random (lower bound).

4. Key Results

Full-model performance: modularity at no cost

| Model | Data | MC9 | Gen5 | MMLU | MMLU-Pro | GSM8K | |:-----|:---------|:---:|:----:|:----:|:--------:|:-----:| | OLMoE† | 5T | 63.5 | 57.6 | 42.8 | 18.7 | 13.7 | | Standard MoE | 1T | 63.9 | 59.7 | 42.4 | 19.3 | 13.9 | | EMO | 1T | 63.1 | 57.9 | 42.8 | 18.5 | 12.0 |

EMO matches standard MoE full-model performance (MMLU 42.8 vs 42.4); the modularization objective has no observable negative impact.

Selective expert use: an order-of-magnitude gap (MMLU, 1T tokens)

| Expert subset | EMO (no FT) | Standard MoE (no FT) | EMO (FT) | Standard MoE (FT) | |:-------------|:-----------:|:----------------:|:-----------:|:----------------:| | 128 (full) | 42.8 | 42.4 | 43.6 | 43.0 | | 64 (50%) | 42.5 | 39.4 | 43.3 | 40.5 | | 32 (25%) | 41.4 | 31.1 | 41.7 | 33.5 | | 16 (12.5%) | 39.9 | 24.6 | 40.1 | 28.4 | | 8 (6.25%) | 36.1 | ~10 | 37.3 | ~10 |

Keeping 25% of experts: EMO drops only 1.4 points (3.3%); standard MoE drops 11.3 (26.7%).
Keeping 12.5%: EMO drops 2.9 (6.8%); standard MoE drops 17.8 (42.0%).
EMO's 32-expert subset (~3.5B total params) outperforms a standard MoE trained from scratch with 32 experts and a dense model of equal memory — breaking the assumption that small memory budgets require training small models from scratch.

Robustness to expert selection

| Method | 32 experts | 16 experts | 8 experts | |:-----|:-------:|:-------:|:------:| | Standard MoE + Router | 31.1 | 24.6 | ~10 | | Standard MoE + Easy-EP | 33.0 | 27.0 | ~10 | | EMO + Router | 41.4 | 39.9 | 36.1 | | EMO + Easy-EP | 41.4 | 39.7 | 36.5 | | Random | ~10 | ~10 | ~10 |

EMO is insensitive to the selection method (<0.5 point gap between Router-based and Easy-EP) — modularity is an intrinsic structural property learned during training, not something post-hoc pruning can recover.

5. Emergent Semantic Specialization

Token clustering (12K documents, PCA to 95% variance, spherical k-means, 32 clusters):

| Model | Top clusters | Granularity | |:-----|:-------------|:---------| | Standard MoE | Prepositions (5.1%), copula verbs (3.8%), definite articles (3.7%) | Syntactic/lexical | | EMO | Film & book reviews (5.1%), health & medical (4.1%), source code (3.4%) | Semantic/domain |

Standard MoE scatters tokens of one document across 10+ clusters; EMO concentrates them in 1–2 clusters, showing within-document consistency and cross-document discrimination.

Domain similarity (24 WebOrganizer domains, cosine similarity of expert-activation vectors): standard MoE's inter-domain similarities span 0.60–0.95 with no clear structure at any layer; EMO's range widens to 0.15–0.85, and deeper layers (10–15) form intuitive groupings (software↔electronics, health↔biology) — semantic specialization emerges progressively with depth.

6. Limitations and Open Questions

| Limitation | Evidence | Impact | |:-----|:---------|:---------| | Weak on "Other" category | 32-expert subset < Dense@8 | Modular advantage vanishes on generic/mixed tasks | | Relies on validation data | Few-shot samples needed for expert selection | Limited zero-shot cold start | | Pretraining only | SFT/RLHF effects unknown | Alignment may destroy modular structure | | Document homogeneity assumption | No cross-domain documents handled | Real web pages often mix topics | | Subset fine-tune re-insertion | Improves overall but below standalone subset | Optimal modular-update protocol unexplored |

The authors interpret the "Other" weakness positively — as evidence EMO truly localizes capabilities, since "Other" is by definition domain-less. Whether this trade-off is acceptable depends on the application.

7. Implications

1. Memory–accuracy Pareto frontier: EMO subsets sit above standard MoE and dense baselines at all tested scales, requiring no model-architecture or inference-engine changes. 2. Behavioral auditability: beyond "what experts do," EMO enables "what the model uses" — activation patterns deviating from the expected expert cluster signal anomalies. 3. Modular content governance: EMO identifies clusters like "spam, adult, gambling & low-quality" (4.1%); disabling expert clusters at inference is reversible and precise compared to data filtering. 4. Modular continual learning: preliminary experiments show fine-tuning a 32-expert subset and re-inserting it improves overall performance, suggesting domain-wise incremental updates without full retraining.

8. Conclusion

EMO demonstrates that a simple document-level constraint suffices to induce semantic-level expert modularity during pretraining, with no negative impact on full-model performance. The deeper implication transcends MoE: a model's functional structure can be shaped by small training-time constraints rather than only post-hoc pruning, distillation, or modular reorganization. Document boundaries are a free, naturally occurring weak-supervision signal — an extremely simple yet effective paradigm for modular deep learning.

Key observations for the next six months:

Scalability verification at >100B total parameters
Whether SFT/RLHF preserves or destroys the modular structure
Transfer of document-boundary constraints to non-English, non-web corpora
Industry adoption (DeepSeek, Qwen, Llama) of similar mechanisms

Paper Information

| Item | Detail | |:-----|:-------| | Title | EMO: Pretraining Mixture of Experts for Emergent Modularity | | Authors | Ryan Wang (UC Berkeley), Akshita Bhagia (Allen Institute for AI), Sewon Min (UC Berkeley & Ai2) | | arXiv | 2605.06663 (2026-05-07, cs.CL / cs.AI) | | Scale | 1B active / 14B total, 128 experts (127 routed + 1 shared), 8 per token | | Data | 1T tokens (OLMoE corpus) + 50B linear annealing | | Paper | https://arxiv.org/abs/2605.06663 | | Code | https://github.com/allenai/EMO | | Models | https://huggingface.co/allenai/EMO | | Visualization | https://emovisualization.netlify.app |