Paper Overview
Research Area: Machine Learning Authors: Minbin Huang, Han Shi, Chuanyang Zheng arXiv: 2505.03485
Abstract
Modern Mixture-of-Experts (MoE) architectures allocate expert capacity through a rigid per-layer rule: each transformer layer owns a separate expert set. This convention couples depth scaling with linear expert-parameter growth and assumes that every layer needs isolated expert capacity. However, recent analyses and the authors' routing probe challenge this allocation rule: replacing a deeper layer's learned top-k router with uniform random routing drops downstream accuracy by only 1.0–1.6 points across multiple production MoE models.
Motivated by this redundancy, the authors propose UniPool, an MoE architecture that treats expert capacity as a global architectural budget by replacing per-layer expert ownership with a single shared pool accessed by independent per-layer routers. To enable stable and balanced training under this sharing mechanism, they introduce:
- A pool-level auxiliary loss that balances expert utilization across the entire pool
- NormRouter, which provides sparse and scale-stable routing to the shared expert pool
- Trained five LLaMA-style models at 182M, 469M, 650M, 830M, and 978M parameters on 30B tokens from The Pile
- UniPool consistently outperforms matched vanilla MoE baselines on validation loss and perplexity, reducing validation loss by up to 0.0386 at these scales
- Slimmed UniPool variants using only 41.6%–66.7% of the vanilla expert-parameter budget match or outperform per-layer MoE at tested scales
- This shows expert parameters need not grow linearly with depth under a shared-pool design; they can scale sub-linearly while being more efficient and effective than vanilla MoE
- Further analysis shows UniPool's advantages compose with finer-grained expert decomposition
Key Results
*Auto-collected on 2026-05-09. Source: zhichai.net forum post.*