Imagine managing a company with 128 departments, each fully staffed, even though every project only uses 4-5 of them. That's the deployment dilemma of Mixture-of-Experts (MoE) models: GPT-OSS-20B has 32 experts per layer and Qwen3-30B has 128, yet each token only activates 4-8 of them. Computation is efficient, but memory is wasteful—all experts must stay resident in VRAM regardless of usage. A 30B-parameter MoE model runs only ~3B parameters of computation per token, yet requires 30B worth of memory.
The obvious fix is pruning unimportant experts—but how do you define "unimportant"?
The blind spot of existing methods
Most prior MoE pruning methods borrow local heuristics from dense Transformer pruning: expert selection frequency, parameter norms, or token distribution coverage. These score each expert independently, ignoring the cross-layer dependencies of MoE routing.
Example: if expert A at layer 5 is activated, expert B at layer 6 may be activated with 80% probability. A and B form a functional pair—pruning either breaks the processing path. Yet viewed layer-by-layer, each may look individually unimportant.
MAESTRO's core idea: routing as a Markov chain
MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based Routing), from IIT Delhi and NVIDIA (arXiv:2607.08601), models each token's activation trajectory across experts as an ergodic Markov chain:
1. State space: All L×E expert slots (L layers × E experts) are chain states; a token traversing layers 1→L is a state sequence. 2. Transition matrices: A small calibration set is run through autoregressive inference (not teacher-forcing), counting co-occurrences of "expert i → expert j" between adjacent layers to get per-layer matrices C^(l). 3. Ergodicity guarantee: The last layer connects back to the first (irreducibility), and self-loops are added (aperiodicity), ensuring a unique stationary distribution. 4. Stationary distribution: Solving for π gives each expert's long-run routing visitation probability. 5. Pruning: Experts with the lowest stationary probability contribute least and are pruned first.
The key insight: the stationary distribution is derived from global trajectories and inherently encodes cross-layer dependencies. Experts that are rarely selected directly but serve as critical intermediate bridges get higher stationary probabilities and are retained.
Engineering details
- Autoregressive calibration vs. teacher-forcing: MAESTRO insists on autoregressive rollouts because teacher-forcing inputs (ground-truth sequences) differ from the model's actual inference distribution, biasing transition statistics. The ablation shows single-pass forward instead of autoregressive rollout costs 1.15% average performance and increases cross-task variance by 10.53%.
- Uniform pruning constraint: Pruning the same fraction per layer, rather than global selection, avoids over-pruning some layers into bottlenecks—trading flexibility for stability, which experiments show is worth it.
- Recovery Fine-Tuning (RFT): A lightweight post-pruning fine-tune is critical: without RFT, average relative performance drops 20.59% and cross-task variance explodes by 1289%.
- 50% compression: MAESTRO achieves up to 10.61% higher average performance retention than the strongest baseline, with significantly lower cross-task variance.
- 25% compression on GPT-OSS-20B: MAESTRO beats the original uncompressed model on 4 tasks—Wikitext perplexity 2.35 vs 2.52, Lambada 1.38 vs 1.42, PIQA 81.18 vs 80.69, CommonsenseQA 80.43 vs 79.12. Pruning helps—a regularization effect.
- At 50% compression: Lambada perplexity is 1.46, only 0.04 above the original—no other method comes close.
- Vs. HC-SMoE: The clustering-based method collapses at 50% compression, dropping to 74.58% average retention with 16.60% standard deviation.
Results
Tested on GPT-OSS-20B (32 experts/layer) and Qwen3-30B (128 experts/layer) across 17 tasks in 5 domains (including safety, bias, and ethics evaluations):
Why is MAESTRO's variance lower?
Local heuristics sometimes accidentally keep task-critical experts and sometimes prune them, causing erratic performance. MAESTRO's global view retains "bridge" experts that benefit all tasks, yielding stable cross-task performance.
Commentary
MAESTRO's elegance lies in converting an engineering question (which experts to cut) into a probability-model question (the Markov chain stationary distribution). Expert importance gets a precise mathematical definition: not "how often selected," but "what share of the long-run routing distribution."
More broadly: MoE experts are not independent parts but nodes in a routing network. A node's value depends on its role in network connectivity, echoing "betweenness centrality" in social network analysis—some nodes carry little traffic, but removing them breaks the network.
RFT's necessity is notable: even the smartest pruning perturbs the output distribution. Without RFT, MAESTRO still beats REAP without RFT by ~1%, showing the pruning itself is higher quality—but RFT's gains are an order of magnitude larger. Compression and recovery are orthogonal problems you must solve together.
Finally, beating the original model at 25% compression suggests over-parameterization: some experts may be memorizing training noise rather than learning useful patterns. MoE architectures may have a "just right" expert count—more can hurt.
---
Paper: https://arxiv.org/abs/2607.08601 HTML full text: https://arxiv.org/html/2607.08601