EMO: Training Mixture-of-Experts Models That Split Like LEGO Bricks

> *"If you can't explain it simply, you don't understand it well enough."* — Richard Feynman

The Problem: A Classroom Full of Geniuses

Imagine a classroom with 128 top specialists — math, code, law. A student asks a calculus question, and only 8 experts answer. But all 128 still occupy the room and consume resources. Worse, if you want to open a branch campus needing only law knowledge, school rules forbid taking just the law experts — everyone must stay together.

This is exactly how standard Mixture-of-Experts (MoE) models behave. MoE was designed to fix the cost problem of trillion-parameter models (like rumored GPT-4 or Kimi K2): instead of one monolithic bronze statue, the feed-forward layers become 128 "expert" blocks, and a router activates only ~8 experts per token — a theoretical 16× efficiency gain.

But experts in standard MoE don't specialize by domain. Studies show they split along low-level syntactic features: "The" goes to expert 12, "cat" to expert 47 — not because of domain expertise, but activation preferences for token patterns. It's like a hospital assigning patients by surname initials instead of by medical department. The consequence: pruning to 25% of experts devastates performance, and 12.5% makes the model nearly unusable. The experts are entangled — remove any one and unexpected input paths collapse.

EMO: Letting Experts Self-Organize

EMO (Emergent Modularity) adds one training constraint to standard next-token-prediction pretraining:

> All tokens within the same document must choose experts from the same shared expert pool.

A document is a Wikipedia article on linear algebra, a Python tutorial, a legal judgment — whatever the pretraining unit is. No domain labels are predefined. But this constraint creates selection pressure: tokens from similar-domain documents gradually converge on similar expert subsets, and modularity emerges spontaneously during training — like dinner guests from the same invitation clustering at the same tables.

Why doesn't standard MoE do this? Because the training objective (cross-entropy on next-token prediction) has no pressure toward domain-level specialization. Syntax-level routing can be a local optimum. EMO doesn't change the objective — it adds a lightweight structural constraint that makes domain-clustered groupings generalize better across documents. It's classic emergent design: simple local rules producing complex global structure, like ants building nests via pheromone-following.

The Numbers: From Collapse to Robustness

The authors pretrained an EMO model on 1 trillion tokens: 14B total parameters, 1B activated per token, matched against a standard MoE of identical size.

| Experts retained | EMO drop | Standard MoE drop | |---|---|---| | 50% (64) | ~0% | significant | | 25% (32) | ~1% | steep fall | | 12.5% (16) | ~3% | nearly unusable | | 6.25% (8) | still usable | collapse |

On MMLU's 16 categories, EMO subsets push the Pareto frontier on the memory-accuracy tradeoff — at any memory budget, EMO subsets beat pruned standard MoE and even small models trained from scratch with the same budget. This makes modular deployment practical:

• A "math-only" version carrying 16 math-focused experts
• A "code-only" version with 8 code experts
• Composable hybrid subsets (math + code)

How Do the Experts Divide the Work?

Analysis of routing logs shows EMO experts form semantic-level clusters: groups specializing in math/logic, code, biomedical text, and law. Per-task subsets vary: coding tasks need 62 experts for near-full performance, web development only 8, biomedical 34. None of this was prescribed — it grew from the constraint. Math-friendly experts get repeatedly selected for math documents, reinforcing stable clusters across thousands of documents.

Ablations: It's Not Luck

1. vs. ModuleFormer (Shen et al., 2023), which maximizes token–expert mutual information: reproduction shows no advantage over standard MoE and sharp degradation below 40% retained experts. EMO drops only 3% at 12.5%. 2. vs. small models trained from scratch: EMO subsets leverage the full 14B model's 1T-token pretraining knowledge, clearly outperforming parameter- and memory-matched baselines. 3. vs. random expert subsets: random selection collapses — the structured selection is essential.

Why It Matters

AI deployment faces a core tension: models grow more general, but deployment scenarios are specific. Hospitals need medical-heavy AI; coding assistants need code-heavy AI. Standard MoE forces loading all 128 experts regardless. As the paper puts it:

> "EMO enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting."

EMO also demonstrates prior-free emergent modularity: no one defines which experts are math or law experts. As new domains appear, models may grow new expert clusters without architectural redesign.

A Feynman Moment

Naming isn't understanding. Why does modularity emerge? What governs its speed and stability? Do domain clusters overlap, and what does the overlap represent? The authors offer a strong first exploration, but these questions remain open — between knowing a phenomenon's name and understanding it lies a universe.

References

1. Ryan Wang et al. "EMO: Pretraining Mixture of Experts for Emergent Modularity." arXiv:2605.06663, 2026. 2. DeepSeek-AI et al. "DeepSeek-V3 Technical Report." arXiv:2412.19437, 2024. 3. Team Olmo. "Olmo 3." 2026. 4. Shazeer et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." ICLR, 2017. 5. Shen et al. "ModuleFormer: Modularity emerges from mixture-of-experts." 2023. 6. Fedus et al. "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity." JMLR, 2022. 7. Shi et al. "FlexOlmo: Open language models for flexible data use." NeurIPS, 2025.

*Written 2026-05-12. Source: arXiv 2026-05-07, via Papers.Cool daily recommendations.*