EMO: Emergent Modularity in Pretrained MoE — A Paradigm Shift via Document-Level Expert Pools

Forum topic · 小凯 · 2026-05-08

Summary

EMO (Emergent Modularity via pretraining MoE), proposed by researchers from UC Berkeley and the Allen Institute for AI, introduces a simple document-level expert pool constraint to Mixture-of-Experts (MoE) pretraining: all tokens within a document must route to a dynamically sampled Top-d subset of experts, with renormalized routing probabilities and global load balancing. Trained on 1T tokens from the OLMoE corpus with 1B active / 14B total parameters (127 routed experts + 1 shared, Top-8 routing), EMO matches standard MoE full-model performance (MMLU 42.8 vs 42.4) while enabling selective expert deployment: keeping only 25% of experts costs about 1 point of MMLU versus roughly 10-11 points for standard MoE. Token clustering and domain-similarity analyses show expert specialization shifts from syntactic (prepositions, articles) to semantic, domain-level (e.g., medical, source code, reviews). EMO subsets even outperform dense models trained at matching memory budgets, and results are robust to expert selection methods. Code, models, and visualizations are open-sourced. Paper: https://arxiv.org/abs/2605.06663.

EMO: Pretraining Emergent Modularity — A Paradigm Shift for MoE Architectures

> TL;DR: UC Berkeley and the Allen Institute for AI propose EMO, which achieves emergent modularity in MoE pretraining through a document-level expert pool constraint. 1B-active / 14B-total parameter models are trained on 1T tokens; keeping only 25% of experts degrades performance by just ~1% (standard MoE degrades ~10%), and expert specialization shifts from the syntactic to the semantic level. Code, models, and visualizations are all open-sourced.

---

1. Problem Reframing: The "Sparsity Paradox" of MoE

Mixture-of-Experts (MoE) has become a core pillar of LLM scaling since Shazeer et al. (2017). Frontier systems like DeepSeek-V3, Qwen3, and Grok-1 all use MoE, promising dense-model expressivity at far lower compute via sparse activation.

> Annotation: Formal definition of MoE > > In a standard MoE layer, a token's hidden state $$x_t$$ passes through router $r(\cdot)$ : >

p_t = \text{softmax}(r(x_t)) \in \mathbb{R}^{n_r}

> where $$n_r$$ is the number of routable experts. The activated set is: >

R_t = \text{TopK}(p_t, k)

> and the output is a weighted combination: >

\text{FFN}_{\text{out}}(x_t) = \sum_{i \in R_t} (p_t)_i \cdot E_i(x_t) + \sum_{j=1}^{n_s} E_j^{(s)}(x_t)

> The second term is a shared expert (always active), typically $$n_s = 1$$ .

However, there is a fundamental deployment-level paradox:

| Dimension | Theoretical promise | Actual deployment | |:----------|:--------------------|:------------------| | Compute | Activate only $$k$$ experts | ✓ Achieved | | Memory | Load only active experts | ✗ Must load all $$n_r + n_s$$ experts | | Specialization | Experts split by task/domain | ✗ Experts split by syntactic role |

Prior work shows standard MoE expert specialization stays at a surface syntactic level — prepositions, punctuation, articles, subword fragments. Consequently, even a purely mathematical task activates most experts, since every token seeks its own "grammar expert."

The direct consequence: MoE sparsity holds only in forward compute, not in memory footprint. As models scale toward trillions of parameters, VRAM becomes the primary MoE deployment bottleneck.

---

2. EMO's Method: Document Boundaries as Weak Supervision

EMO's core contribution elevates modularity to a first-class pretraining objective. The method is minimal: use naturally occurring document boundaries in the pretraining corpus to constrain all tokens in a document to share one expert pool.

2.1 Document-Level Expert Pool Constraint

Step 1: Compute the document-level average routing distribution:

\bar{p} = \frac{1}{T} \sum_{t=1}^{T} \text{softmax}(r(x_t))

Step 2: Select the Top- $$d$$ experts as the document pool:

D = \text{TopK}(\bar{p}, d), \quad D \subseteq \{1, \ldots, n_r\}

Step 3: All tokens in the document must choose activated experts from $$D$$ :

\hat{p}_t(i) = \begin{cases} \dfrac{(p_t)_i}{\sum_{j \in D} (p_t)_j} & \text{if } i \in D \\[6pt] 0 & \text{otherwise} \end{cases}

R_t = \text{TopK}(\hat{p}_t, k)

> Annotation: Why renormalization matters > > Zeroing out-of-pool experts without renormalizing makes routing probabilities sum to < 1, decaying gate weights. EMO's conditional renormalization ensures $\sum_{i \in D} \hat{p}_t(i) = 1$ , preserving the weighted sum over expert combinations.

2.2 Dynamic Pool Size via Randomization

A fixed pool size $$d$$ would make the model rigid at inference. EMO instead samples $d \sim \mathcal{U}\{k, k+1, \ldots, n_r\}$ per document, exposing the model to all possible subset sizes during pretraining so it stays robust across memory budgets.

2.3 Global Load Balancing

Document-level constraints conflict with standard micro-batch load balancing, which aggregates few documents and tends to scatter a document's tokens across experts. EMO uses global load balancing across data-parallel groups:

\mathcal{L}_{\text{LB}} = n_r \sum_{i=1}^{n_r} \bar{f}_i \cdot \bar{P}_i

where $\bar{f}_i$ is the fraction of tokens routed to expert $$i$$ and $\bar{P}_i$ its average routing probability. Cross-document balancing and intra-document pool sharing are complementary, not conflicting.

---

3. Setup

| Configuration | Value | |:--------------|:------| | Active parameters | 1B | | Total parameters | 14B | | Routed experts $$n_r$$ | 127 | | Shared experts $$n_s$$ | 1 | | Experts per token $$k$$ | 8 | | Pretraining data | 1T tokens (OLMoE corpus) | | Annealing | 50B tokens (linear) | | Baseline | Same-architecture standard MoE, same data/scale |

Evaluation covers full-model zero-shot (MC9, Gen5, MMLU, MMLU-Pro, GSM8K) and selective expert use, comparing Router-based selection, Easy-EP (SOTA expert pruning), and Random (lower bound).

---

4. Key Results

4.1 Full-Model Performance: Modularity at Zero Cost

| Model | Data | MC9 | Gen5 | MMLU | MMLU-Pro | GSM8K | |:------|:-----|:---:|:----:|:----:|:--------:|:-----:| | OLMoE† | 5T | 63.5 | 57.6 | 42.8 | 18.7 | 13.7 | | Standard MoE | 1T | 63.9 | 59.7 | 42.4 | 19.3 | 13.9 | | EMO | 1T | 63.1 | 57.9 | 42.8 | 18.5 | 12.0 |

† OLMoE uses an older architecture (no pre-norm, QK-norm, no shared expert, micro-batch balancing), 64 experts.

EMO at 1T tokens matches standard MoE (MMLU 42.8 vs 42.4) and both beat OLMoE trained on 5T. The modularity objective has no observable negative effect on full-model performance.

4.2 Selective Expert Use: Orders-of-Magnitude Gap

| Expert subset | EMO (no FT) | Standard MoE (no FT) | EMO (with FT) | Standard MoE (with FT) | |:--------------|:-----------:|:--------------------:|:-------------:|:----------------------:| | 128 (full) | 42.8 | 42.4 | 43.6 | 43.0 | | 64 (50%) | 42.5 | 39.4 | 43.3 | 40.5 | | 32 (25%) | 41.4 | 31.1 | 41.7 | 33.5 | | 16 (12.5%) | 39.9 | 24.6 | 40.1 | 28.4 | | 8 (6.25%) | 36.1 | ~10 (random) | 37.3 | ~10 |

(MMLU macro-averaged, excluding "other"; 1T-token training.)

| Metric | EMO | Standard MoE | |:-------|:---:|:------------:| | MMLU drop at 25% experts | -1.4 (3.3%) | -11.3 (26.7%) | | MMLU drop at 12.5% experts | -2.9 (6.8%) | -17.8 (42.0%) | | MMLU drop at 6.25% experts | -6.7 (15.7%) | ~32 (~75%) |

Deeper implication: EMO's 32-expert subset (~3.5B total params) outperforms both a from-scratch 32-expert standard MoE and dense models of equal memory — a subset extracted from a large pretrained model beats a small model trained specifically for that budget.

4.3 Robustness to Expert Selection

| Method | 32 experts | 16 experts | 8 experts | |:-------|:---------:|:---------:|:---------:| | Standard MoE + Router | 31.1 | 24.6 | ~10 | | Standard MoE + Easy-EP | 33.0 | 27.0 | ~10 | | EMO + Router | 41.4 | 39.9 | 36.1 | | EMO + Easy-EP | 41.4 | 39.7 | 36.5 | | Random | ~10 | ~10 | ~10 |

EMO is insensitive to selection method (< 0.5 points difference), showing modularity is an endogenous structural property learned during training, not a surface recoverable by post-hoc pruning.

---

5. Emergent Semantic Specialization

5.1 Token Clustering

Tokens from 12K documents were embedded via routing probabilities, PCA-reduced (95% variance), L2-normalized, and clustered (spherical k-means, 32 clusters):

| Model | Top clusters | Granularity | |:------|:-------------|:------------| | Standard MoE | "Prepositions" (5.1%), "Copula verbs" (3.8%), "Definite articles" (3.7%) | Syntactic/lexical | | EMO | "Film & book reviews" (5.1%), "Health & medical" (4.1%), "Source code" (3.4%) | Semantic/domain |

In standard MoE, tokens from the same document scatter across 10+ clusters; in EMO they concentrate in 1-2 — direct evidence of within-document consistency and cross-document distinctiveness.

5.2 Domain Similarity Matrix

Using 24 human-annotated domains from WebOrganizer:

| Feature | Standard MoE | EMO | |:--------|:------------:|:---:| | Inter-domain similarity range | 0.60 – 0.95 | 0.15 – 0.85 | | Related-domain clustering | No clear structure | software ↔ electronics, health ↔ biology | | Unrelated-domain separation | Fuzzy | Clear | | Across layers | Chaotic at all depths | Shallow chaos → deep structure |

EMO's deeper layers (10–15) exhibit human-intuitive domain groupings, suggesting semantic specialization emerges progressively, with deeper layers gradually "recognizing" document domains.

---

6. Limitations and Open Questions

| Limitation | Evidence | Potential impact | |:-----------|:---------|:-----------------| | Weak "Other" category | 32-expert subset < Dense@8 | Modular advantage vanishes on general/mixed tasks | | Relies on validation data | Few-shot samples needed to select experts | Limited zero-shot cold start | | Pretraining only | SFT/RLHF effects unknown | Downstream alignment may destroy modularity | | Document homogeneity assumption | No cross-domain documents handled | Real web content mixes topics | | Subset fine-tune reinsertion | Improves but below standalone subset | Optimal modular-update protocol TBD |

Notably, the authors interpret the weak "Other" category positively — as evidence that "EMO genuinely learns localized abilities," since "Other" is by definition domain-less. Whether this trade-off is acceptable depends on whether the goal is general or domain-specific capability.

---

7. Impact Assessment

EMO's modular architecture opens five independent impact pathways:

7.1 Restructuring the Memory-Accuracy Pareto Frontier

Existing MoE deployment research focuses on memory-constrained scaling laws, expert pruning, and dynamic offloading. EMO offers an orthogonal path: no model or inference-engine changes — just subset selection. EMO subsets sit above the Pareto frontier of standard MoE and dense baselines at all tested scales.

7.2 Capability-Level Interpretability

Standard MoE interpretability asks "what do experts do?" EMO adds "what does the model use?" (behavioral auditing): if a math question's activation pattern deviates from the math expert cluster, that is an anomaly signal — more actionable than attention visualizations or neuron probing.

7.3 Modular Content Governance

EMO's clusters clearly identify a "spam, adult, gambling & low-quality" cluster (4.1%), inspiring governance at inference time: disable specific expert clusters instead of filtering data. Expert disabling is reversible and precise, with theoretically lower false-positive rates.

7.4 Modular Continual Learning

Preliminary experiments: fine-tuning a 32-expert subset and reinserting it improves overall performance (though not to standalone-subset levels). If matured, this enables per-domain incremental updates — train new-domain expert subsets independently and integrate them without retraining the full model.

---

8. Conclusion

EMO's core methodological contribution is proving that: a simple document-level constraint suffices to induce semantic-level expert modularity during pretraining, with no negative effect on full-model performance.

This finding suggests a more general proposition:

> A model's functional structure can be shaped by small training-time constraints, rather than only through post-hoc pruning, distillation, or modular reorganization.

EMO's constraint (document boundaries) is a free weak-supervision signal native to pretraining corpora — an "exploit existing structure rather than introduce new annotation" paradigm for modular deep learning.

Key things to watch in the next six months: 1. Scalability validation at larger scale (>100B total parameters) 2. Whether SFT and RLHF preserve or destroy modular structure 3. Cross-corpus transfer: document-boundary constraints on non-English, non-web corpora 4. Industry adoption: whether next-gen MoEs from DeepSeek, Qwen, Llama integrate similar mechanisms

---

📚 Paper Details

| Item | Content | |:-----|:--------| | Title | EMO: Pretraining Mixture of Experts for Emergent Modularity | | Authors | Ryan Wang (UC Berkeley), Akshita Bhagia (Allen Institute for AI), Sewon Min (UC Berkeley & Ai2) | | Institutions | UC Berkeley, Allen Institute for AI | | arXiv ID | 2605.06663 | | Published | 2026-05-07 | | Categories | cs.CL, cs.AI | | Key claim | Document-level expert pool constraints induce emergent semantic modularity in MoE pretraining, enabling independent deployment and composition of expert subsets with no full-model performance loss | | Scale | 1B active / 14B total parameters, 128 experts (127 routed + 1 shared), Top-8 per token | | Training data | 1T tokens (OLMoE corpus) + 50B linear annealing | | Key result | Keeping 25% of experts costs ~1%, keeping 12.5% costs ~3%; standard MoE drops ~10% and ~15% under the same conditions | | Paper | https://arxiv.org/abs/2605.06663 | | Code | https://github.com/allenai/EMO | | Models | https://huggingface.co/allenai/EMO | | Visualization | https://emovisualization.netlify.app |