UniPool: A Globally Shared Expert Pool for Mixture-of-Experts

Forum topic · 小凯 · 2026-05-10

Summary

UniPool is a new Mixture-of-Experts (MoE) architecture that replaces per-layer expert ownership with a single globally shared expert pool, accessed by each Transformer layer through its own router. The design is motivated by routing-probe experiments showing that replacing learned top-k routers with uniform random routing in production MoE models degrades downstream accuracy by only 1.0-1.6 points, revealing redundancy in per-layer expert allocation. To stabilize training, UniPool introduces a pool-level auxiliary loss for balanced expert utilization and NormRouter for sparse, scale-stable routing. Trained on 30B tokens of the Pile across five LLaMA-architecture model sizes (182M-978M parameters), UniPool consistently improves validation loss and perplexity over vanilla MoE baselines, with up to 0.0386 lower validation loss. Notably, pool size acts as an explicit depth-scaling hyperparameter: pruned-pool variants using only 41.6%-66.7% of vanilla expert parameter budgets match or exceed per-layer MoE performance, demonstrating that expert parameters can grow sub-linearly with depth. The advantages also compose with finer-grained expert decomposition.

Paper Overview

Research Area: ML Authors: Minbin Huang, Han Shi, Chuanyang Zheng, Yimeng Wu et al. Published: 2026-05-07 arXiv: 2605.06665

Abstract

Modern Mixture-of-Experts (MoE) architectures allocate expert capacity under a strict layer-wise rule: each Transformer layer owns its own set of experts. This convention couples depth scaling with a linear increase in expert parameters, assuming every layer requires independent expert capacity. However, recent analyses and the authors' routing probes challenge this allocation rule: across multiple production-grade MoE models, replacing learned top-k routers in deeper layers with uniform random routing degrades downstream accuracy by only 1.0-1.6 percentage points.

Motivated by this redundancy, the authors propose UniPool, an MoE architecture that treats expert capacity as a global architectural budget — replacing per-layer expert ownership with a single shared expert pool, where each layer accesses the pool via its own independent router. To enable stable and balanced training under this sharing scheme, they introduce:

A pool-level auxiliary loss that balances expert utilization across the entire pool
NormRouter, providing sparse and scale-stable routing into the shared expert pool

Results

UniPool consistently improves validation loss and perplexity compared to matched vanilla MoE baselines
At these scales, UniPool reduces validation loss by up to 0.0386 versus vanilla MoE
Pool size acts as an explicit depth-scaling hyperparameter: pruned-pool UniPool variants using only 41.6%-66.7% of the vanilla expert parameter budget match or exceed per-layer MoE performance
This shows expert parameters need not grow linearly with depth — they can grow sub-linearly while remaining more efficient and effective than vanilla MoE
Further analysis shows UniPool's advantages compose with finer-grained expert decomposition

--- *Auto-collected on 2026-05-10*