The Empty Castle: Cutting Half of MoE Experts Can Actually Speed Up Inference
🏃♂️ Introduction: The Bloated "Brain Trust"
Today's top AI models love playing the "strength in numbers" game. This architecture is called MoE (Mixture-of-Experts). Although only a few experts work at any moment, that long row of "advisors" standing there makes routing and data movement exhausting. Worse, for simple tasks that the AI could handle with its eyes closed, it still wakes up the big experts — using a sledgehammer to crack a nut.
Can the model skip the experts for easy inputs and travel light?
🔬 The Problem: Easy to Summon, Hard to Dismiss
Teaching an already-trained MoE model to "slack off" is not easy. Previous approaches either require pretraining from scratch or task-specific adaptation. For already-trained giant models, changing their work habits often scrambles their brains, causing large drops in intelligence. This is the "inertia of static architecture": after training, every expert believes itself indispensable, occupying its slot even for menial jobs.
> 💡 Tip: The "Inference Cost Bottleneck" refers to the huge waste of GPU memory and compute caused by the model being forced to activate a preset number of experts during inference.
⚖️ The Breakthrough: ZEDA's "Empty Castle" Strategy
In May 2026, the ZEDA framework arrived to "downsize" MoE models. It doesn't demolish anything — it performs a trick of "creating something from nothing":
1. Inject a "null expert": secretly add a zero-output dummy expert to each MoE layer. 2. Two-stage self-reflection: the model learns, by repeatedly studying its own frozen self (as teacher), when to hand work to the null expert — "zero-expert self-distillation." 3. Dynamic routing: it instantly identifies "easy" tokens, sends them through the empty slot, and saves the real experts for hard problems.
The optimization logic is captured by this compact formula:
> 💡 Note: The routing space now includes an all-zero expert (\(\mathbf{0}\)). Whenever the input (\(x\)) is simple enough, the AI picks this "empty slot," saving substantial FLOPs.
ZEDA's Downsizing Results
| Dimension | Traditional Static MoE | ZEDA Dynamic | Verdict | | :--- | :--- | :--- | :--- | | Compute (FLOPs) | Full load | Cut by half (50%+) | Leaner | | Inference speed | Slow and steady | 1.2x faster | Nimble | | Capability retention | 100% (baseline) | Near-lossless | Slimmer, not dumber |
📈 Field Results: Not Just Faster, But Steadier
The results leave legacy dynamic-MoE algorithms behind. On top models like Qwen3 and GLM-4.7, ZEDA not only sheds half the expert load but also scores 4–6 points higher than prior methods on hardcore benchmarks in math and code. It proves one thing: AI must learn not only to be "knowledgeable," but also to "conserve effort."
📝 References
- Paper title: Post-Trained MoE Can Skip Half Experts via Self-Distillation
- Published: May 19, 2026
- Identifier: arXiv:2605.18643
- Core contribution: Solves the redundant computation problem of pre-trained static MoE models at inference time, enabling low-cost dynamic architecture conversion.
- Mechanism: The ZEDA framework injects zero-output experts and applies two-stage self-distillation, allowing the model to skip expert activation for simple inputs.