EMO: How Emergent Modularity Makes Giant MoE AI Models Dismantlable Like Lego

Forum topic · 小凯 · 2026-05-11

Summary

This post explains EMO (Emergent Modularity), a new training approach for Mixture-of-Experts (MoE) language models. Standard MoE models route tokens to experts based on low-level syntactic features rather than semantic domains, so experts cannot be selectively removed: retaining only 25% of experts causes severe performance collapse. EMO adds a lightweight constraint during pretraining: all tokens from the same document must select experts from a shared expert pool. Without any predefined domain labels, this creates selection pressure that makes experts spontaneously cluster by semantic area (math, code, biomedicine, law). The authors pretrained a 14B-parameter (1B activated per token) EMO model on 1 trillion tokens and showed that keeping 25% (12.5%) of experts causes only a 1% (3%) absolute performance drop, while standard MoE breaks at the same settings. EMO subsets also outperform parameter- and memory-matched models trained from scratch, and beat ModuleFormer-style mutual-information objectives. This enables modular deployment: domain-specific subsets (e.g., 16 experts for coding) can be extracted from one large pretrained model, pushing the memory-accuracy Pareto frontier on MMLU. The post includes analogies, ablation details, and discussion of why emergence occurs without priors.

When Experts Learn to Cluster: How EMO Makes Giant AI Models Dismantlable Like Lego

> *"If you can't explain something simply, you don't truly understand it."* — Richard Feynman

This is an English translation/summary of a Chinese forum post analyzing the paper "EMO: Pretraining Mixture of Experts for Emergent Modularity" (Ryan Wang et al., arXiv:2605.06663).

🌌 A Classroom Full of "Geniuses"

Imagine a classroom with 128 top specialists in different fields. When someone asks a calculus question, only 8 help — but all 128 sit there consuming resources. And if you want to move just the law experts to a branch campus, you can't: school rules require everyone to attend together, or knowledge gaps appear.

That classroom is a picture of standard Mixture-of-Experts (MoE) models: models with total parameters in the trillion-parameter range (e.g., rumored GPT-4 scale, Kimi K2), where a router activates only a few experts (e.g., 8 of 128) per token, but the whole model must still be loaded as one indivisible "bronze statue."

🎭 The "Split Personality" of Experts

Researchers examining standard MoE expert specialization found experts do not divide by domain (math, code, law). Instead, they specialize by low-level syntactic features — e.g., one expert responds to tokens starting with "Th," another to monosyllabic nouns.

It's like a hospital where doctors are assigned by patients' surname initials rather than by specialty. The fatal consequence: even with 128 experts, you cannot truly use just a subset. Experiments show retaining 25% of experts in a standard MoE causes severe performance collapse; 12.5% is nearly unusable.

🌿 EMO: Letting Experts Self-Organize

EMO's core training constraint:

> All tokens within the same document must select experts from the same shared expert pool.

A "document" is a pretraining unit: a linear-algebra Wikipedia article, a Python tutorial, a legal judgment. No pools are predefined — but during training, an emergent phenomenon appears: tokens from similar-domain documents converge to similar expert subsets. The authors call this Emergent Modularity: modularity that grows naturally during training rather than being engineered.

Why doesn't standard MoE do this? Its objective is next-token prediction, which has no pressure toward domain-level division of labor — syntactic-level routing is a perfectly good local optimum for prediction accuracy. EMO doesn't change the objective; it adds a lightweight structural constraint that creates selection pressure favoring domain-clustered expert groups — a classic case of emergent design via simple local rules.

📊 The Data: From Collapse to Robustness

Setup: a 14B-total-parameter (1B activated per token) EMO model pretrained on 1 trillion tokens, compared against a size-matched regular MoE. Then experts were pruned:

| Experts retained | EMO drop | Standard MoE drop | |---|---|---| | 50% (64) | ~0% | significant | | 25% (32) | 1% | severe | | 12.5% (16) | 3% | nearly unusable | | 6.25% (8) | still usable | collapse |

On MMLU's 16 categories, EMO subsets push the memory-accuracy Pareto frontier, beating standard MoE subsets and even small models trained from scratch at the same memory budget. This makes modular deployment real: deploy a "math-only" 16-expert version, an 8-expert code version, or compose subsets.

🧠 What Do the Experts "Talk" About?

Analysis of routing logs shows EMO experts form semantic specializations: clusters for math/logic, code/programming, biomedical text, and law. E.g., coding tasks need only 62 experts to approach full-model performance; web development needs 8; biomedicine needs 34. No one told the model which experts should handle which domains — it learned this itself under the document-level pool constraint.

🔬 Ablations: Not Luck

vs. ModuleFormer (Shen et al., 2023), which maximizes token–expert mutual information: it shows no advantage over standard MoE and degrades sharply below 40% of experts retained; EMO drops only 3% at 12.5%.
vs. small models trained from scratch: EMO subsets leverage full 14B-pretrained knowledge and win at matched memory.
vs. random expert subsets: random selection collapses — EMO's expert use is structured, not lucky.

🌍 Why It Matters

AI deployment's core tension: models are increasingly general, but deployment scenarios are specific (hospitals need medicine, coding assistants need code). Standard MoE forces loading everything; EMO lets you "bring only the necessary people to the necessary occasion":

> "EMO enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting."

It also demonstrates prior-free emergent modularity — no predefined domain-to-expert mapping, suggesting models could grow new expert clusters as new domains appear.

🎭 A Feynman Moment: Naming ≠ Understanding

The post closes with self-reflection: saying "emergent modularity happened" is not explaining *why* it emerges, what governs its speed and stability, or what overlapping clusters across domains mean. Open questions for follow-up research.

📚 References

1. Ryan Wang et al. "EMO: Pretraining Mixture of Experts for Emergent Modularity." arXiv:2605.06663, 2026. 2. DeepSeek-AI et al. "DeepSeek-V3 Technical Report." arXiv:2412.19437, 2024. 3. Team Olmo. "Olmo 3." 2026. 4. Shazeer et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." ICLR, 2017. 5. Shen et al. "ModuleFormer: Modularity emerges from mixture-of-experts." 2023. 6. Fedus et al. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." JMLR, 2022. 7. Shi et al. "FlexOlmo: Open language models for flexible data use." NeurIPS, 2025.

*Originally published 2026-05-12; paper via Papers.Cool daily picks (arXiv 2026-05-07).*