UniPool: A Globally Shared Expert Pool That Breaks the Layer-Isolation Barrier in MoE Models

Forum topic · 小凯 · 2026-05-09

Summary

UniPool replaces the per-layer private expert sets of standard Mixture-of-Experts (MoE) Transformers with a single globally shared expert pool. The authors first show via a routing probe that replacing learned top-k routers with uniform random routing in deep MoE layers only drops downstream accuracy by 1.0-1.6 points, suggesting expert capacity is largely redundant across layers. UniPool addresses the challenges of shared routing with a pool-level auxiliary load-balancing loss (preventing expert collapse) and NormRouter (LayerNorm before routing scores for scale-consistent selection). Across five LLaMA-style models (182M-978M parameters, 30B Pile tokens), UniPool consistently lowers validation loss (up to -0.039) and improves average downstream accuracy by roughly 0.8-1.9 points on seven benchmarks (ARC, PIQA, HellaSwag, WinoGrande, LAMBADA, RACE). Notably, pool size can scale sub-linearly with depth: UniPool matches or exceeds standard MoE with 50% or less of the expert parameter budget, and combining fine-grained FFN decomposition with the shared pool yields further gains. This reframes pool size as an explicit depth-scaling hyperparameter rather than one tied to layer count.

UniPool: When All Experts Share One Pool — Ending MoE's Layer-Isolation Barrier

> "Every Transformer layer gets its own experts? That's like building a swimming pool on every floor — UniPool asks: why not share one?"

This post reviews UniPool: A Globally Shared Expert Pool for Mixture-of-Experts (Huang et al., arXiv:2605.06665), which challenges the default assumption that each Transformer layer in an MoE model needs its own private expert set.

The Problem: Layer-Isolated Expert Resources

Standard MoE architecture (e.g., Switch Transformer, DeepSeekMoE) assigns a dedicated set of FFN experts to each layer. In a 32-layer model with 8 experts per layer, experts in layer 1 only serve layer 1, layer 2's only serve layer 2, and so on — a total of 256 expert slots. The design assumption is that different layers handle different abstraction levels (syntax at lower layers, semantics/reasoning at higher layers) and therefore need independent expert capacity.

The Routing Probe Finding

The authors' motivating experiment: in several production-scale MoE models, replace the learned top-k routers in deep layers with uniform random routing. Surprisingly, downstream task accuracy dropped by only 1.0-1.6 percentage points.

This suggests deep-layer experts are barely input-discriminative, and expert capacity is largely redundant across layers — maintaining a full private expert pool per layer is wasteful.

The UniPool Design

UniPool replaces per-layer expert sets with one globally shared expert pool; every layer's router selects top-k experts from the same pool. Two key components make this work:

1. Pool-Level Auxiliary Loss — a load-balancing objective applied over the entire pool (L_aux = PoolBalance(f_1, ..., f_M)), preventing expert collapse where a few experts get overused by all layers. 2. NormRouter — LayerNorm applied before computing routing scores, so routers from different layers (with very different input distributions) produce comparably scaled scores, avoiding certain layers dominating popular experts.

Experimental Results

Evaluated on 5 LLaMA-style model scales (182M-978M), trained on 30B tokens of the Pile:

Validation loss: consistently lower than standard MoE at all scales (improvements of 0.023-0.039; largest at 830M).
Downstream benchmarks (ARC-Easy, ARC-Challenge, PIQA, HellaSwag, WinoGrande, LAMBADA, RACE): average gains of +0.75 to +1.85 points, peaking at the 830M scale (43.82 → 45.67).

Sub-Linear Expert Scaling

The deepest contribution: expert parameter count no longer needs to grow linearly with depth. With a 32-layer × 8-expert baseline, UniPool variants match or exceed baseline with:

| Config | Expert budget (relative) | Performance | |---|---|---| | Standard MoE | 100% | baseline | | UniPool (Pool=16) | ~50% | matches or exceeds | | UniPool (Pool=12) | ~37.5% | near baseline | | UniPool (Pool=8) | ~25% | still competitive |

Pool size thus becomes an explicit depth-scaling hyperparameter instead of being dictated by layer count.

Fine-Grained Expert Decomposition

Combining the shared pool with finer FFN decomposition into micro-experts yields further gains at 182M scale: 38.74 (standard MoE) → 39.61 (UniPool) → 41.22 (UniPool + 32 micro-experts).

Why Sharing Works

Parameter reuse: a capability learned by one layer's expert (e.g., handling math symbols) is directly available to all layers, instead of being relearned.
Larger routing space: each layer chooses from a larger global pool despite fewer total parameters.
Training stability: pool-level balancing and NormRouter keep shared routing stable.

Broader Implications

UniPool suggests the assumption that layers must own private computational resources is at least partially wrong. If experts can be shared across layers, layers may function as "processing stages" rather than "resource boundaries" — pointing toward future architectures organized more like an expert network than a stack of expert layers.

References

1. Huang, M., et al. (2026). UniPool: A Globally Shared Expert Pool for Mixture-of-Experts. *arXiv preprint arXiv:2605.06665*. 2. Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers. *JMLR, 23*(120), 1-39. 3. DeepSeek-AI (2024). DeepSeekMoE. *arXiv:2401.06066*. 4. Shazeer, N., et al. (2017). Outrageously Large Neural Networks. *ICLR 2017*. 5. Jacobs, R. A., et al. (1991). Adaptive Mixtures of Local Experts. *Neural Computation, 3*(1), 79-89. 6. Krajewski, J., et al. (2024). Scaling Laws for Fine-Grained Mixture of Experts. *arXiv:2402.07871*. 7. Zheng, C., et al. (2025). Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel. *arXiv:2509.25913*.