EMO: Pretraining MoE Where Experts Emerge as Lego-like Domain Modules

Forum topic · 小凯 · 2026-05-08

Summary

EMO (Emergent Modularity via pretraining MoE), from researchers at UC Berkeley and the Allen Institute for AI, introduces a minimal change to standard Mixture-of-Experts training: all tokens within a document must select experts from a shared document-level expert pool, sampled at random sizes during pretraining. This self-supervised constraint, requiring no domain labels or classifiers, causes experts to specialize by semantic domain (math, code, biomedical, reviews) rather than by surface syntax like prepositions and punctuation, as happens in standard MoE. A 1B-active/14B-total parameter model with 128 experts trained on 1T OLMoE tokens retains 41.4 MMLU with only 32 experts (vs 42.8 full) and 39.9 with 16 experts, while a standard MoE drops to 31.1 and 24.6 respectively. A fine-tuned 8-expert EMO subset fully recovers GSM8K performance. The post reviews results, limitations (weaker on catch-all 'other' tasks, validation data needed for expert selection, untested after instruction tuning), and applications such as on-device deployment, content filtering, modular updates, and interpretability.

> My first mental image after reading this paper: someone took a 14B-parameter monster and split it into 128 Lego bricks. Each brick is functional — math, code, biomedical, film reviews, each in its place. Even wilder: you only need 16 of them to do 90% of the original job.

This isn't science fiction. It's EMO (Emergent Modularity via pretraining MoE) by Ryan Wang, Akshita Bhagia, and Sewon Min, from UC Berkeley and the Allen Institute for AI.

1. MoE's "Emperor's New Clothes"

Mixture-of-Experts has been hyped endlessly. DeepSeek-V3, Qwen3, Grok — these trillion-parameter beasts all run on MoE, marketed as: "we only activate a few experts, so it's sparse, efficient, memory-friendly."

> Annotation: MoE (Mixture of Experts) > > Replace each Transformer FFN layer with $$N$$ small networks ("experts"), plus a router that picks the top- $$k$$ experts per token: >

\text{MoE}(x) = \sum_{i \in \text{TopK}(k)} g_i(x) \cdot E_i(x)

> where $$g_i(x)$$ is the router's gating weight and $$E_i(x)$$ is expert $$i$$ 's output.

The catch — you still have to fit the whole model in memory. Even for a pure math task, a standard 128-expert MoE ends up activating 120+ experts to some degree. Why? Because experts don't divide by "math/code/medicine." They divide by prepositions, punctuation, articles:

> Standard MoE's top clusters: "Word-internal subword fragments" (6.0%), "Prepositions of/in/for/to/on/with" (5.1%), "Copula verbs is/are/was/be" (3.8%)...

It's like a company claiming "on-demand staffing," but when you ask accounting to do the books, receptionists and security also show up — because they own "filling forms" and "opening doors."

That's not sparsity. That's schizophrenic density.

2. EMO's Breakthrough: Document Boundaries

The insight is beautiful in its simplicity — no fancy domain classifier, no manual labels. Just one rule:

> All tokens in the same document must pick experts from the same expert pool.

> Annotation: document-level expert pool constraint > > Standard MoE: each token independently picks its Top- $$k$$ experts: >

p_t = \text{softmax}(r(x_t)) \in \mathbb{R}^{n_r}, \quad R_t = \text{TopK}(p_t, k)

> > EMO: average the routing distribution over a document's $$T$$ tokens and pick a Top- $$d$$ document pool $$D$$ : >

D = \text{TopK}\left(\frac{1}{T}\sum_{t=1}^T p_t, \; d\right)

> All tokens in the document choose only from $$D$$ . Pool size $$d$$ is randomly sampled during training: >

d \sim \mathcal{U}\{k, k+1, \ldots, n_r\}

> So the model has seen every subset size, enabling flexible pruning at inference.

Why do document boundaries suffice? Because tokens in the same document most likely come from the same domain — a math paper doesn't suddenly contain a recipe (unless it's about "optimal cooking temperature," which is math too 🍳).

The constraint is self-supervised: no domain labels, no classifier, no human priors. Pretraining corpora naturally come with document boundaries.

3. The Numbers Don't Lie

EMO's scale: 1B active / 14B total parameters, 128 experts (127 routed + 1 shared), 8 active per token. Trained from scratch on 1T tokens of OLMoE corpus plus 50B linear annealing.

| Deployment | EMO | Standard MoE | Gap | |:-----------|:---:|:------------:|:---:| | Full model (128 experts) | 42.8 MMLU | 42.4 MMLU | ≈ even | | 25% experts (32) | 41.4 | 31.1 | +10.3 | | 12.5% experts (16) | 39.9 | 24.6 | +15.3 | | 6.25% experts (8) | 36.1 | ~10 (random) | +26 |

> 📊 From the paper's Figure 3 and Table 1, 1T-token setting, no fine-tuning.

EMO with only 32 experts drops just 1% on MMLU (42.8 → 41.4). Standard MoE drops 11.3% (42.4 → 31.1).
EMO's 32-expert subset outperforms a standard 32-expert MoE trained from scratch.

Meaning: no need to retrain a separate small model for low-memory deployment — just pull out a quarter of EMO's experts. 😲

On GSM8K it's even starker. EMO's 8-expert subset, after fine-tuning, fully recovers full-model performance. Standard MoE's 8-expert subset outputs this:

> Reg. MoE, 8-expert subset: "Olivia $2005 2005 2005 2005 2005 2005 2005 ... [+162 more] 200"

Gibberish. 🤯 EMO's 8-expert subset:

> EMO, 8-expert subset: "Harry slept 9 hours. James slept 2/3 of 9 = 6. 9 - 6 = 3. The answer is 3." ✅

4. Experts Learn "Domains," Not "Grammar"

The paper's Figure 5 visualization is brilliant: PCA + spherical k-means over routing behavior, 32 clusters named by Claude Code.

Standard MoE clusters:

"Word-internal subword fragments" (6.0%)
"Prepositions" (5.1%)
"Abstract nouns & domain topics" (4.8%)
"Section breaks & boilerplate punctuation" (4.6%)

EMO clusters:

"Film, music, TV & book reviews" (5.1%)
"Health, medical & wellness" (4.1%)
"Tech support, IT ops & web dev" (4.1%)
"Source code" (3.4%)
"U.S. politics & elections" (3.2%)
"Biomedical & life-sciences research" (3.1%)

Standard MoE experts learn syntactic roles — tokens scatter across "grammar experts." EMO experts learn semantic domains — the model learns "this is a medical article, use the medical experts." Using WebOrganizer's 24 human-labeled domains, EMO's expert-activation similarity matrix clearly clusters related domains (software_development correlates with electronics_and_hardware), while standard MoE's matrix is a fog (all domain similarities > 0.6).

5. Don't Pop the Champagne Yet

Some caveats. 😐

1. EMO's "other" category is weak. On the catch-all "other" MMLU/MMLU-Pro category, EMO's 32-expert subset loses to a Dense@8 model trained from scratch. Modularity dilutes general-purpose ability — the authors themselves exclude "other" from aggregate MMLU reporting. 2. Expert selection needs validation data. The paper says a few-shot example suffices, but you must know the task; a totally unseen domain may not map to a subset. 3. Pretraining only. Whether instruction tuning, RLHF, or tool use destroys the modularity is untested. Mixed-domain SFT data could blur the clean division of labor. 4. Are document boundaries enough? Real web documents aren't always single-domain; a blog can be half technical, half personal. Forcing all tokens into one pool may hurt on highly mixed content — not deeply discussed by the authors.

6. What Does It Mean?

> Standard MoE is an apartment building with 128 residents; ring any doorbell and 8 people come help — but they're organized by "who's good at opening doors" and "who's good at filling forms," not by profession. EMO is an office building organized by floor: 3rd floor finance, 5th floor medical, 7th floor IT. Know what you need, go to the right floor; you don't light up the whole building.

The analogy breaks on boundary-blurring tasks (e.g., a digital finance system for hospitals spans finance + IT + medical) — hence EMO's weak "other" performance. Even so, the potential is large:

| Use case | Possibility | |:---------|:------------| | 📱 On-device | Load 16 experts, run a 14B-class model on a phone | | 🧒 Content filtering | Just switch off "spam/gambling/adult" expert clusters | | 🏥 Domain specialization | Load only biomedical experts for medical QA | | 🔧 Modular updates | Fine-tune one expert group, plug it back in | | 🔍 Interpretability | Expert activation reveals which domain the model is "thinking about" |

That last point is especially interesting: if a math answer activates "creative writing" or "low-quality web" experts, that's a red flag 🚩. Expert activation becomes a structured behavioral audit signal — far more concrete than squinting at attention maps.

7. Closing Thoughts

EMO's biggest impact isn't the numbers (though they're gorgeous) — it's the proof that a model's structure doesn't have to be monolithic. As the authors put it:

> *"Large language models need not remain monolithic systems. Modularity can be built into pretraining itself."*

In plain terms: the monster need not exist forever in monster form. It can be a pack of wolves that can be split apart and regrouped at will. 🐺

The bet: if nobody replicates this in six months, EMO is just a pretty proof of concept. But given open-sourced models, code, and visualization tools, and near-zero engineering overhead — I'd bet it works. And I'd bet DeepSeek's and Qwen's next-gen MoEs are already quietly trying this. 👀

---

📚 Paper Details

| Item | Content | |:-----|:--------| | Title | EMO: Pretraining Mixture of Experts for Emergent Modularity | | Authors | Ryan Wang (UC Berkeley), Akshita Bhagia (Allen Institute for AI), Sewon Min (UC Berkeley & Ai2) | | Institutions | UC Berkeley, Allen Institute for AI | | arXiv ID | 2605.06663 | | Published | 2026-05-07 | | Categories | cs.CL, cs.AI | | Model scale | 1B active / 14B total, 128 experts (127 routed + 1 shared), 8 active per token | | Training data | 1T tokens (OLMoE corpus) + 50B linear annealing | | Key result | 25% of experts: only −1%; 12.5%: only −3%; standard MoE: −10% and −15% respectively | | Paper | https://arxiv.org/abs/2605.06663 | | Code | https://github.com/allenai/EMO | | Models | https://huggingface.co/allenai/EMO | | Visualization | https://emovisualization.netlify.app |