Paper Overview
Research area: Machine Learning Authors: Minbin Huang, Han Shi, Chuanyang Zheng Published: 2026-05-06 arXiv: 2505.03485
Summary
Modern Mixture-of-Experts (MoE) architectures allocate expert capacity through a rigid per-layer rule: each transformer layer owns a separate expert set. This convention couples depth scaling with linear expert-parameter growth and assumes that every layer needs isolated expert capacity. However, recent analyses and the authors' routing probe challenge this allocation rule: replacing a deeper layer's learned top-k router with uniform random routing drops downstream accuracy by only 1.0-1.6 points across multiple production MoE models.
Key Contributions
- UniPool architecture: Expert capacity is treated as a global architectural budget. A single shared expert pool replaces per-layer expert ownership, with each layer accessing the pool through its own independent router.
- Pool-level auxiliary loss: Balances expert utilization across the entire shared pool, enabling stable and balanced training under the sharing mechanism.
- NormRouter: Provides sparse and scale-stable routing into the shared expert pool.
- Trained on 30 billion tokens from The Pile across five LLaMA-architecture model sizes: 182M, 469M, 650M, 830M, and 978M parameters.
- UniPool consistently outperforms matched vanilla MoE baselines on both validation loss and perplexity, reducing validation loss by up to 0.0386 at these scales.
- Slimmed-pool UniPool variants using only 41.6%-66.7% of the vanilla expert parameter budget match or outperform per-layer MoE at the tested scales.
- This demonstrates that expert parameters need not grow linearly with depth under a shared-pool design; they can scale sub-linearly while being more efficient and effective than vanilla MoE.
- Further analysis shows UniPool's advantages stack with finer-grained expert decomposition.
Experimental Results
Significance
The results identify pool size as an explicit depth-scaling hyperparameter, challenging the long-standing assumption that every transformer layer requires its own isolated expert capacity in MoE models.