ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation

Forum topic · 小凯 · 2026-06-20

Summary

ZEDA is a post-training adaptation framework that converts static Mixture-of-Experts (MoE) language models into dynamic ones by injecting zero-computation "zero experts" into each MoE layer. Using a two-stage self-distillation pipeline (supervised fine-tuning followed by on-policy distillation) plus a group auxiliary loss that balances load only between the normal-expert and zero-expert groups, the router learns to skip experts adaptively without modifying any original expert parameters. Applied to Qwen3-30B-A3B and GLM-4.7-Flash, ZEDA achieves over 50% zero-expert activation (51.2% and 53.0% respectively) with minimal accuracy loss (74.2 vs 74.9 and 71.8 vs 72.5 average across 11 benchmarks) and roughly 20% inference speedup in both prefill and decode. Adaptation costs only 31 hours on 8 H200 GPUs for Qwen3-30B-A3B using 60k prompts, orders of magnitude cheaper than pretraining a dynamic MoE from scratch. The framework outperforms baselines like AdaMoE and Dynamic Skipping, and analysis shows zero-expert activation correlates with token-level uncertainty rather than task-level difficulty. Code and paper (arXiv: 2605.18643) are available from TsinghuaC3I, Kuaishou, and Shanghai AI Lab collaborators.

ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation

ZEDA is a post-training adaptation framework from TsinghuaC3I, Frontis.AI, Kuaishou, Shanghai AI Lab, and WeChat AI that turns an already post-trained static MoE model into a dynamically routed one — at a fraction of the cost of training a dynamic MoE from scratch.

Paper: *Post-Trained MoE Can Skip Half Experts via Self-Distillation* (arXiv: 2605.18643)
Code: https://github.com/TsinghuaC3I/ZEDA
Team: Xingtai Lv, Li Sheng, Kaiyan Zhang, et al. (TsinghuaC3I, Frontis.AI, Kuaishou, Shanghai AI Lab, WeChat AI)

Key points

Problem: Static MoE models (e.g., Qwen3-30B-A3B, GLM-4.7-Flash, DeepSeek-V3) activate a fixed number of experts per token, wasting computation on easy tokens. Existing dynamic MoE methods (MoE++, LongCat, AdaMoE) require pretraining from scratch or task-specific retraining, making them unusable for expensive, already post-trained models.
Core idea: Inject zero experts — experts whose output is always 0, adding no parameters or computation — into each MoE layer's routing candidate pool. When the router selects a zero expert, that expert slot's computation is skipped.
Zero experts vs copy experts: Copy experts (identity output) introduce scale and direction mismatches that destroy performance (AIME24 accuracy drops from 80.9% to 1.0% on Qwen3-30B-A3B). Zero experts are true no-ops that leave the residual stream untouched.
Two-stage self-distillation: Stage 1 is SFT on teacher-generated data to stabilize routing with the enlarged candidate pool; Stage 2 is on-policy distillation (OPD) on the student's own rollouts to fix the off-policy distribution mismatch. Both stages are necessary.
Group auxiliary loss: Load balancing is enforced only *between* the normal-expert group and the zero-expert group, preserving the carefully tuned non-uniform routing among original experts. With group weight w = 2, the target zero-expert activation rate is ~50% — the Pareto-optimal operating point.
No router renormalization: Renormalizing surviving expert weights after a zero expert is selected consistently hurts accuracy, since the original model's top-K weight magnitudes are already calibrated.

Results

On Qwen3-30B-A3B (128 experts, top-8), averaged over 11 benchmarks:

| Method | Avg accuracy | Zero-expert rate | AIME24 | AIME25 | GSM8k | MATH-500 | LCBv5 | IFEval | |---|---|---|---|---|---|---|---|---| | Original | 74.9 | 0% | 80.9 | 71.0 | 95.4 | 94.4 | 61.5 | 86.3 | | AdaMoE | 54.8 | 51.9% | 25.0 | 24.8 | 92.4 | 79.8 | 36.1 | 82.4 | | Dynamic Skipping | 68.1 | 43.8% | 78.1 | 67.9 | 95.2 | 94.4 | 57.3 | 70.4 | | ZEDA | 74.2 | 51.2% | 79.0 | 69.1 | 95.5 | 95.2 | 58.2 | 84.3 |

On GLM-4.7-Flash (64 experts, top-4), ZEDA reaches 71.8 average (vs 72.5 original) with 53.0% zero-expert activation.

Efficiency: ~20% speedup in both prefill and decode at 8k sequence length using SGLang.
Cost: 31 hours on 8x H200 for Qwen3-30B-A3B; 62 hours for GLM-4.7-Flash. Training uses 60k prompts (17k math, 15k code, 28k conversation), saturating around 60k samples.
Ablations: ZEDA (74.2) beats SFT-only (73.3) and direct expert-halving variants (72.3–73.0), confirming both the zero-expert mechanism and the OPD stage matter.

What does zero-expert activation learn?

Zero-expert rate correlates negatively with token entropy and teacher-student logp-diff: uncertain tokens use more experts, confident tokens use fewer.
Code and math expressions show higher zero-expert rates than natural language, reflecting stronger structural regularity.
Activation rate is roughly constant across MATH-500 difficulty levels — allocation is token-level, not task-level.

Limitations

Validated only at ~30B scale; 100B+ untested.
No evaluation on long-horizon agent tasks or multimodal MoE.
Speedup slightly decays beyond 8k context.

Bottom line

ZEDA doesn't train a better MoE — it teaches an existing one *when to skip work*. For production MoE models (Qwen3, GLM-4, DeepSeek-V3), it offers a near-lossless acceleration path at minimal cost: 8 GPUs, ~31 hours, skip half the experts, keep the accuracy.

Key parameters: group weight w = 2, loss coefficient α = 0.1, no router renormalization.

Reference info

arXiv: 2605.18643
GitHub: https://github.com/TsinghuaC3I/ZEDA
Benchmarks: AIME24/25/26, GSM8k, MATH-500, LCBv5/v6, HumanEval+, MBPP+, IFBench, IFEval (11 total)

ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation

Summary

ZEDA: Post-Trained MoE Models Can Skip Half Their Experts via Self-Distillation

Key points

Results

What does zero-expert activation learn?

Limitations

Bottom line

Tags