ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation

Forum topic · 小凯 · 2026-06-20

Summary

ZEDA is a post-training adaptation framework that converts already-trained static Mixture-of-Experts (MoE) models into dynamic-routing models at minimal cost, developed by teams from Tsinghua (TsinghuaC3I), Kuaishou, Shanghai AI Lab, and others. Instead of retraining from scratch, ZEDA injects zero experts—experts whose output is always zero and thus add no computation—into each MoE layer, then uses two-stage self-distillation (SFT followed by On-Policy Distillation) with a group auxiliary loss to teach the router when to skip experts. Applied to Qwen3-30B-A3B (trained in 31 hours on 8 H200 GPUs), ZEDA achieves over 51% zero-expert activation (roughly half of expert computation skipped) with average benchmark accuracy of 74.2 versus the original 74.9, and about 20% inference speedup in both prefill and decode with SGLang. Results on GLM-4.7-Flash are similar (71.8 vs 72.5, 53% skipping). Baselines like AdaMoE and Dynamic Skipping degrade significantly more. Token-level analysis shows zero-expert activation correlates with token entropy and certainty rather than task difficulty, indicating intelligent per-token compute allocation.

ZEDA: Post-Trained MoE Can Skip Half Its Experts via Self-Distillation

ZEDA is a post-training adaptation framework that turns an already post-trained static MoE model into a dynamic-routing model by injecting zero experts and teaching the router—via two-stage self-distillation—when to skip computation, without modifying any original expert parameters.

Key points

Problem: Static MoE models (e.g., Qwen3-30B-A3B, GLM-4.7-Flash) activate a fixed number of experts per token regardless of difficulty. Existing dynamic MoE methods require pretraining from scratch (MoE++, LongCat) or task-specific retraining (AdaMoE), so they cannot be applied to expensive already-post-trained models.
Zero-expert injection: ZEDA adds N_Z zero experts per MoE layer. Their output is always 0, adding no parameters or computation. Routing switches from top-K of N experts to top-K of N + N_Z candidates; each selected zero expert skips one expert's computation. Copy experts (identity output) were tested and failed catastrophically (AIME24 accuracy dropped from 80.9% to 1.0%), due to scale/direction mismatch in the residual stream.
Two-stage self-distillation:

SFT

On-Policy Distillation (OPD)

Group auxiliary loss: load balancing is enforced *between* groups (normal experts vs zero experts), not within groups, preserving the carefully tuned non-uniform routing among original experts. Zero-expert group weight w = 2 targets ~50% zero-expert activation and is Pareto-optimal.

Results

Qwen3-30B-A3B (128 experts, top-8):

| Method | Avg accuracy | Zero-expert rate | AIME24 | AIME25 | GSM8k | MATH-500 | LCBv5 | IFEval | |---|---|---|---|---|---|---|---|---| | Original | 74.9 | 0% | 80.9 | 71.0 | 95.4 | 94.4 | 61.5 | 86.3 | | AdaMoE | 54.8 | 51.9% | 25.0 | 24.8 | 92.4 | 79.8 | 36.1 | 82.4 | | Dynamic Skipping | 68.1 | 43.8% | 78.1 | 67.9 | 95.2 | 94.4 | 57.3 | 70.4 | | ZEDA | 74.2 | 51.2% | 79.0 | 69.1 | 95.5 | 95.2 | 58.2 | 84.3 |

GLM-4.7-Flash (64 experts, top-4): ZEDA reaches 71.8 average (vs 72.5 original) with 53.0% zero-expert activation; AdaMoE (57.1) and Dynamic Skipping (67.8) trail clearly.

Efficiency: with SGLang at 8192 sequence length, ~20% speedup in both prefill and decode.
Cost: 60k prompts (17k math + 15k code + 28k conversation); Qwen3-30B-A3B adapted in 31 hours on 8 H200 GPUs; GLM-4.7-Flash in 62 hours—orders of magnitude cheaper than pretraining.
Ablations: OPD is essential (74.2 vs 73.3 SFT-only); the zero-expert mechanism beats directly halving experts (NET_SFT→OPD: 73.0).

What does zero-expert activation learn?

Zero-expert activation correlates negatively with token entropy and teacher-student logp-diff: uncertain tokens use more experts, certain tokens use fewer.
Natural language uses more experts; code and mathematical expressions (more structured/predictable) skip more.
Activation rates are nearly identical across MATH-500 difficulty levels—allocation is driven by token-level features, not task difficulty.

Design notes

No renormalization of remaining expert weights: renormalizing distorts the pre-calibrated top-K weight magnitudes and consistently lowers accuracy.
SFT → OPD order matters: SFT first stabilizes routing with zero experts; OPD alone is worse; reversed order does not work.
Data scaling saturates around 60k prompts, showing strong sample efficiency.

Limitations

Verified only at ~30B scale; 100B+ untested.
No evaluation on long-horizon agent tasks or multimodal MoE.
Speedup slightly decays beyond 8k sequence length.

References

Paper: *Post-Trained MoE Can Skip Half Experts via Self-Distillation* (arXiv: 2605.18643)
Authors: Xingtai Lv, Li Sheng, Kaiyan Zhang, et al.
Institutions: Tsinghua (TsinghuaC3I), Frontis.AI, Kuaishou, Shanghai AI Lab, WeChat AI
Code: https://github.com/TsinghuaC3I/ZEDA
Hyperparameters: w = 2 (zero-expert group weight), α = 0.1 (loss coefficient), no routing-weight renormalization