UniPool: A Globally Shared Expert Pool for Mixture-of-Experts

Forum topic · 小凯 · 2026-05-10

Summary

UniPool is a new Mixture-of-Experts (MoE) architecture that replaces per-layer expert ownership with a single globally shared expert pool, where each Transformer layer accesses the pool through its own router. The authors observed that in several production MoE models, replacing learned top-k routers with uniform random routing in deeper layers only reduces downstream accuracy by 1.0-1.6 points, suggesting per-layer expert capacity is largely redundant. To stabilize training under sharing, UniPool introduces a pool-level auxiliary loss balancing expert utilization across the whole pool, plus NormRouter for sparse, scale-stable routing. Trained on 30B tokens from the Pile across five LLaMA-style model sizes (182M-978M parameters), UniPool consistently improves validation loss and perplexity over vanilla MoE baselines, with up to a 0.0386 validation loss reduction. Notably, pruned-pool variants using only 41.6%-66.7% of vanilla expert parameter budgets match or exceed per-layer MoE performance, showing expert parameters can scale sub-linearly with depth. Benefits also combine with finer-grained expert decomposition. Paper: arXiv:2605.06665.

Paper Overview

Field: Machine Learning Authors: Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu et al. arXiv: 2605.06665

Abstract (English)

Modern Mixture-of-Experts (MoE) architectures allocate expert capacity through strict per-layer rules: every Transformer layer owns an independent set of experts. This convention couples depth scaling with linear growth in expert parameters, assuming each layer requires dedicated expert capacity. However, recent analyses and the authors' routing probes challenge this allocation rule: across multiple production-level MoE models, replacing learned top-k routers in deeper layers with uniform random routing degrades downstream accuracy by only 1.0-1.6 percentage points.

Motivated by this redundancy, the authors propose UniPool, an MoE architecture that treats expert capacity as a global architectural budget. UniPool replaces per-layer expert ownership with a single shared expert pool, where each layer accesses the pool via its own independent router.

Key Contributions

Pool-level auxiliary loss: balances expert utilization across the entire shared pool for stable and even training.
NormRouter: provides sparse and scale-stable routing into the shared expert pool.
Sub-linear depth scaling: expert parameters need not grow linearly with model depth under the shared-pool design.

Experimental Results

Trained on 30B tokens from the Pile across five LLaMA-architecture model sizes: 182M, 469M, 650M, 830M, and 978M parameters.
UniPool consistently improves validation loss and perplexity over matched vanilla MoE baselines, reducing validation loss by up to 0.0386 at these scales.
Pruned-pool UniPool variants using only 41.6%-66.7% of the vanilla expert parameter budget match or exceed per-layer MoE performance at tested scales.
Further analysis shows UniPool's advantages are complementary to finer-grained expert decomposition.

Takeaway

Shared expert pools decouple expert capacity from depth: expert parameters can scale sub-linearly while remaining more efficient and effective than vanilla per-layer MoE, and pool size becomes an explicit depth-scaling hyperparameter.

---

*Source: arXiv:2605.06665, auto-collected 2026-05-10.*