> Reading this paper, the first image in my head was: someone disassembled a 14B-parameter beast into 128 Lego bricks. Each brick is not decoration—math, code, biomedicine, film reviews, each in its place. Even wilder, you only need 16 of them to recover ~90% of the original capability.
This is not science fiction. It is EMO (Emergent Modularity via pretraining MoE), released today on arXiv by Ryan Wang, Akshita Bhagia, and Sewon Min from UC Berkeley and the Allen Institute for AI.
---
1. The Emperor's New Clothes of MoE
Mixture-of-Experts (MoE) has been hyped for too long. DeepSeek-V3, Qwen3, Grok—all trillion-parameter monsters run on MoE. The marketing line is always the same: *"We only activate a small fraction of experts, so it's sparse, efficient, memory-friendly."*
> Annotation: Mixture-of-Experts (MoE) > > Replace each Transformer FFN layer with N small neural networks ("experts"), then a router picks the top-k most relevant experts for each input token. Formally: > $$\text{MoE}(x) = \sum_{i \in \text{TopK}(k)} g_i(x) \cdot E_i(x)$$ > where $g_i(x)$ is the gating weight from the router and $E_i(x)$ is the i-th expert's output. > > Why it matters: in theory, only k experts are computed, far cheaper than one giant dense FFN.
The problem is—you still have to load the entire model into VRAM.
The paper uses a painfully vivid example: even if your task is just "solve a math problem," more than 120 of the 128 experts in a standard MoE get activated to varying degrees. Why? Because experts don't partition by "math/code/medicine." They partition by prepositions, punctuation, and articles. 😅
> Top clusters in standard MoE: "Word-internal subword fragments" (6.0%), "Prepositions of/in/for/to/on/with" (5.1%), "Copula verbs is/are/was/be" (3.8%)…
It is like a company that claims to "dispatch on demand," but when you hire an accountant, you get one or two finance staff—plus the receptionist, janitor, and security guard, because they handle "filling forms" and "opening doors," which bookkeeping also happens to need.
That isn't sparsity. That's schizophrenia-flavored density.
---
2. EMO's Breakthrough: Document Boundaries
The Ryan Wang team's insight is so clean I want to slap the table.
They didn't build any fancy domain classifier, and they didn't hand-label "this is math, that's code." They did one thing:
> All tokens within the same document must pick experts from the same expert pool.
That's it.
> Annotation: Document-level expert-pool constraint > > In standard MoE, each token independently picks its Top-k experts: > $$p_t = \text{softmax}(r(x_t)) \in \mathbb{R}^{n_r}$$ > $$R_t = \text{TopK}(p_t, k)$$ > > EMO first averages the routing distribution over the T tokens of a document, selects the top-d experts to form the document expert pool D: > $$D = \text{TopK}\left(\frac{1}{T}\sum_{t=1}^T p_t, \; d\right)$$ > > Then every token in the document picks its k active experts only from this pool D. The pool size d is sampled randomly during training: > $$d \sim \mathcal{U}\{k, k+1, \ldots, n_r\}$$ > > This way the model sees every subset size during training and can be pruned flexibly at inference.
Why are document boundaries enough? Because tokens in the same document almost certainly come from the same domain. A math paper won't suddenly insert a recipe (unless it's about "optimal cooking temperature," which is still math 🍳).
The beauty is that this constraint is self-supervised—no human labels required; pretraining corpora naturally have document boundaries. No domain labels, no classifier, no human priors.
---
3. The Numbers Don't Lie
EMO's scale: 1B active / 14B total parameters, 128 experts (127 routed + 1 shared), 8 active per token. Pretrained from scratch on 1T tokens of OLMoE data plus 50B of linear annealing.
Core results:
| Deployment | EMO | Standard MoE | Gap | |:-----------|:---:|:------------:|:---:| | Full model (128 experts) | 42.8 MMLU | 42.4 MMLU | ≈ tie | | Keep 25% experts (32) | 41.4 | 31.1 | +10.3 | | Keep 12.5% experts (16) | 39.9 | 24.6 | +15.3 | | Keep 6.25% experts (8) | 36.1 | ~10 (random) | +26 |
> 📊 Source: paper Figure 3 and Table 1, 1T-token training, no fine-tuning.
The cruelty of this comparison:
- EMO loses only 1% MMLU when keeping 32 experts (42.8 → 41.4).
- Standard MoE loses 11.3% under the same condition (42.4 → 31.1).
- Even harsher: EMO's 32-expert subset outperforms a standard MoE trained from scratch with 32 experts.
- "Word-internal subword fragments" (6.0%)
- "Prepositions" (5.1%)
- "Abstract nouns & domain topics" (4.8%)
- "Section breaks & boilerplate punctuation" (4.6%)
- "Film, music, TV & book reviews" (5.1%)
- "Health, medical & wellness" (4.1%)
- "Tech support, IT ops & web dev" (4.1%)
- "Source code" (3.4%)
- "U.S. politics & elections" (3.2%)
- "Biomedical & life-sciences research" (3.1%)
What this means: you don't need to retrain a smaller model for low-memory deployment. Just pull 1/4 of the experts out of EMO, and you still beat a purpose-built small MoE. 😲
On GSM8K (math reasoning) it's even more dramatic: after fine-tuning, EMO's 8-expert subset fully recovers the full-model score. Standard MoE's 8-expert subset outputs this:
> Reg. MoE, 8-expert subset: "Olivia $2005 2005 2005 2005 2005 2005 2005 ... [+162 more] 200"
Yes—gibberish. 🤯 While EMO's 8-expert subset produces:
> EMO, 8-expert subset: "Harry slept 9 hours. James slept 2/3 of 9 = 6. 9 - 6 = 3. The answer is 3." ✅
---
4. Experts Learn Domains, Not Grammar
The paper's Figure 5 visualization is a masterstroke. They use PCA + spherical k-means to cluster pretraining tokens by routing behavior into 32 groups, then let Claude Code name each cluster.
Standard MoE clusters:
EMO clusters:
Standard MoE experts learn syntactic constituents—prepositions, verbs, punctuation. Because each token picks experts independently, the model discovers "all prepositions look similar" and routes them to the same experts. Result: tokens in one document get scattered as each hunts for its own "grammar expert."
EMO experts learn semantic domains—math, code, medicine, politics. Because the document-level constraint forces one document to share an expert pool, the model must learn "this is a medical article, so let's use medical experts."
The paper also verifies this: using WebOrganizer's 24 human-labeled domains, they compute inter-domain expert activation similarity. EMO's similarity matrix cleanly clusters related domains (software_development highly correlates with electronics_and_hardware), while standard MoE's matrix is chaos (all-domain similarities > 0.6).
---
5. Don't Pop the Champagne Yet
A few honest caveats. 😐
First, EMO performs poorly on the "other" category. On MMLU and MMLU-Pro's catch-all "other" bucket, EMO's 32-expert subset loses to a Dense@8 model trained from scratch. Modularity has a cost: general capability gets diluted. If you need a "do-everything" small model, EMO subsets aren't necessarily optimal. The authors are upfront—they deliberately exclude "other" from their aggregate MMLU reporting.
Second, expert selection requires validation data. The paper says "a few-shot example is enough," but that still means you have to know your task in advance. Throw an unseen domain at it, and it may not know which Lego bricks to pull.
Third, the paper only covers pretraining. Instruction tuning, RLHF, tool use—do these downstream stages break the modular structure? The authors don't say, but intuitively, if SFT mixes too much cross-domain data, the clean separation learned via document boundaries may blur.
Fourth, and what nags me most: are document boundaries really enough? The web's documents aren't always "single-domain." A blog post may be half technical, half personal reflection. EMO's approach is "the whole document shares one pool," and this coarse constraint may force different-domain tokens into the same pool, hurting performance on highly mixed content. The authors don't dig into this.
---
6. What Does This Actually Mean?
Let me put my cards on the table. 💰
**My take: EMO is not another MoE variant. It is the first model to optimize for *modularity* as a first-class pretraining objective. And its method is absurdly simple—one line of routing logic changed, no extra data, no extra labels.
An analogy:
> Standard MoE is an apartment building with 128 tenants. Every time you ring a bell, 8 people come down to help. The problem: different tasks need different professional skills, but tenants group by "who's good at opening doors" and "who's good at filling forms"—not by "who's an accountant" or "who's a doctor." So you hire an accountant, but the doorman and janitor also show up. > > EMO is an office building zoned by floor: floor 3 is finance, floor 5 is medicine, floor 7 is IT. You know what you need to do, walk to the right floor, grab 8 people. The whole building doesn't need to be lit.
Where does the analogy break? On boundary-blurred tasks. If your task spans "finance + medicine + IT" (say, building a hospital's digital finance system), you may still have to visit several floors. That's why EMO struggles on "other"—"other" is essentially "hard to classify" tasks.
Even so, EMO's implications are large:
| Use case | Possibility | |:---------|:-----------| | 📱 On-device deployment | Load only 16 experts, run a 14B-class model on a phone | | 🧒 Content filtering | Disable the "spam/gambling/adult" expert cluster | | 🏥 Domain specialization | Load only biomedical experts for medical QA | | � Modular updates | Fine-tune one expert group, plug back into the full model | | 🔍 Interpretability | See active experts to know what "domain" the model is thinking about |
The last point is especially interesting. The paper notes: if a model answering a math question activates "creative writing" or "low-quality web" experts, that's a red flag 🚩. Expert activations become a structured behavioral auditing signal—far more concrete than "squinting at attention maps to guess what the model is thinking."
---
7. Closing
What hits me hardest about EMO isn't the numbers (though they're gorgeous). It's the proof that model structure need not be a monolithic slab.** We've been trapped by the "one model, one set of weights, one full load" paradigm, and EMO pries that door open with a near-zero-cost constraint.
The authors close with a sentence I quote in full:
> *"Large language models need not remain monolithic systems. Modularity can be built into pretraining itself."*
In plain language: the beast need not always exist in beast-form. It can be a pack of wolves that scatter and reform at will. �
The risk: if no one replicates a similar result in the next six months, or document-level constraints turn out not to work on other corpora/scales, EMO is just a beautiful proof of concept. But the authors have open-sourced models, code, and visualization tools, and the method itself has almost no engineering barrier.
I bet it works.
And I bet DeepSeek and Qwen's next-generation MoE are already quietly trying this. 👀
---
📚 Paper Details
| Item | Details | |:-----|:--------| | Title | EMO: Pretraining Mixture of Experts for Emergent Modularity | | Authors | Ryan Wang (UC Berkeley), Akshita Bhagia (Allen Institute for AI), Sewon Min (UC Berkeley & Ai2) | | Institutions | UC Berkeley, Allen Institute for AI | | arXiv ID | 2605.06663 | | Released | 2026-05-07 | | Categories | cs.CL (Computation and Language), cs.AI (Artificial Intelligence) | | Core claim | A document-level expert-pool constraint during pretraining lets MoE spontaneously develop semantic-level modularity, enabling expert subsets to be deployed and combined independently without losing full-model performance | | Model scale | 1B active / 14B total params, 128 experts (127 routed + 1 shared), 8 active per token | | Training data | 1T tokens (OLMoE corpus) + 50B linear annealing | | Key results | Keeping 25% experts drops only 1%; keeping 12.5% drops only 3%; standard MoE under the same conditions drops 10% and 15% respectively | | Paper | https://arxiv.org/abs/2605.06663 | | Code | https://github.com/allenai/EMO | | Model | https://huggingface.co/allenai/EMO | | Visualization | https://emovisualization.netlify.app |
#CrushAI #FeynmanLearning #智柴系统实验室 #MoE #MixtureOfExperts #Modularity #EfficientAI #DeepSeek #EMO