Summary
UniPool is a new Mixture-of-Experts (MoE) architecture proposed by Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu et al. (arXiv 2605.06665) that replaces per-layer expert ownership with a single globally shared expert pool accessed by independent routers at each layer. The motivation comes from routing probe experiments showing that replacing learned top-k routers with uniform random routing in deep layers of production MoE models degrades downstream accuracy by only 1.0-1.6 percentage points, suggesting redundant per-layer expert capacity. To stabilize training of the shared pool, UniPool introduces a pool-level auxiliary loss for balanced expert utilization and NormRouter for sparse, scale-stable routing. Trained on 30B tokens of the Pile across five LLaMA-style model sizes (182M to 978M parameters), UniPool consistently improves validation loss and perplexity over vanilla MoE, with up to 0.0386 lower validation loss. Notably, reduced-pool variants using only 41.6%-66.7% of vanilla expert parameter budgets match or exceed per-layer MoE performance, showing expert parameters can scale sub-linearly with depth. Benefits compose with finer-grained expert segmentation.
Overview
- Field: Machine Learning
- Authors: Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu et al.
- Published: 2026-05-07
- arXiv: 2605.06665
Key Points
Modern Mixture-of-Experts (MoE) architectures allocate expert capacity with strict layer-wise rules: each Transformer layer owns an independent set of experts. This convention couples depth scaling with linear growth in expert parameters, assuming every layer requires its own expert capacity.
Motivation from routing probes: Analysis of multiple production-grade MoE models shows that replacing learned top-k routers in deep layers with uniform random routing degrades downstream accuracy by only 1.0-1.6 percentage points — evidence of substantial redundancy in per-layer expert allocation.
UniPool design: Inspired by this redundancy, UniPool treats expert capacity as a global architectural budget. A single shared expert pool replaces per-layer expert ownership, and each layer accesses the pool through its own independent router.
- A pool-level auxiliary loss balances expert utilization across the entire pool for stable, even training.
- NormRouter provides sparse and scale-stable routing into the shared expert pool.
Experimental results: UniPool was trained on 30B tokens from the Pile across five LLaMA-architecture model sizes (182M, 469M, 650M, 830M, and 978M parameters). It consistently improves validation loss and perplexity over matched vanilla MoE baselines, reducing validation loss by up to 0.0386 at these scales.
Sub-linear depth scaling: Pool size acts as an explicit depth-scaling hyperparameter. Reduced-pool UniPool variants using only 41.6%-66.7% of the vanilla expert parameter budget match or exceed per-layer MoE performance at the tested scales. Expert parameters thus need not grow linearly with depth — they can grow sub-linearly while remaining more efficient and effective than vanilla MoE.
Compatibility: Further analysis shows UniPool's advantages can be combined with finer-grained expert segmentation.
---
*Auto-collected on 2026-05-10.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619688