Empty City Strategy: MoE Models Skip Half Their Experts and Get Faster, Not Slower

Forum topic · 小凯 · 2026-05-25

Summary

A Chinese tech forum post introduces ZEDA, a framework that makes post-trained Mixture-of-Experts (MoE) models dynamically skip expert activation for easy inputs. Static MoE models activate a fixed number of experts per token, causing redundant computation. ZEDA injects a zero-output dummy expert into each MoE layer and uses a two-stage self-distillation process (against a frozen teacher) to teach the router when to route simple tokens to the empty expert, formulated as E_dynamic = Route(x, {E1...En} ∪ {0}). On Qwen3 and GLM-4.7 models, ZEDA reportedly cuts FLOPs by over 50%, speeds up inference by 1.2x, and preserves quality with near-zero loss, outperforming prior dynamic MoE methods by 4-6 points on math and coding benchmarks. Source: arXiv:2605.18643, 'Post-Trained MoE Can Skip Half Experts via Self-Distillation' (May 19, 2026).

The Problem: Bloated Expert Teams

Top AI models increasingly rely on Mixture-of-Experts (MoE) architectures. Although only a few experts activate per token, maintaining a long roster of experts makes routing and data movement expensive. Worse, easy tokens that the model could handle trivially still wake up heavyweight experts — using a sledgehammer to crack a nut.

Why Fixing It Is Hard

Previous approaches require either retraining from scratch (pre-training) or task-specific adaptation. For already-trained large MoE models, changing their routing behavior tends to disrupt learned representations and sharply degrade intelligence — a kind of "static architecture inertia": every expert believes it is indispensable, even for trivial work.

> Tip: The "Inference Cost Bottleneck" refers to the massive waste of memory and compute caused by forcing a fixed number of experts to activate at inference time.

The ZEDA Framework

Released May 2026, ZEDA performs a "mass layoff" of experts without destructive surgery:

1. Inject a zero-output expert: a dummy expert producing all-zeros is secretly added to each MoE layer. 2. Two-stage self-distillation: the model learns, against its own frozen version (teacher), when to delegate work to the empty expert — dubbed "zero-expert self-distillation." 3. Dynamic routing: simple tokens are routed to the empty expert, reserving real experts for hard inputs.

The core logic in one equation:

$ E_{dynamic} = \text{Route}(x, \{E_1...E_n\} \cup \{ \mathbf{0} \}) $

This means the routing pool now includes an all-zero expert ( $\mathbf{0}$ ). When the input ( $$x$$ ) is easy enough, the router picks the empty slot, saving substantial FLOPs.

Results

On top models such as Qwen3 and GLM-4.7, ZEDA cut expert overhead in half while scoring 4–6 points higher than prior dynamic MoE methods on math and coding benchmarks — proving that models should learn not just to be knowledgeable, but to economize effort.

References

Paper: *Post-Trained MoE Can Skip Half Experts via Self-Distillation*
Published: May 19, 2026
arXiv: arXiv:2605.18643
Contribution: Solves redundant computation in pre-trained static MoE models via low-cost dynamic architecture conversion using zero-expert injection and two-stage self-distillation.