UniPool: A Globally Shared Expert Pool That Breaks Layer Barriers in MoE Models

> *"Every Transformer layer has its own experts? That's like building a swimming pool on every floor. UniPool asks: why not share one?"*

The Problem: Per-Layer Expert Isolation

In standard Mixture-of-Experts (MoE) architectures, each Transformer layer maintains its own private set of experts (FFN modules). A 32-layer model with 8 experts per layer dedicates each layer's experts exclusively to that layer, on the assumption that different layers need independent expert capacity because they process different levels of abstraction (syntax at lower layers, semantics and reasoning at higher layers).

UniPool's authors questioned this assumption.

The Routing Probe Experiment

The motivating finding: using a routing probe, the authors replaced the learned top-k routers in deep layers (e.g., layer 20) of production-level MoE models with uniform random routing. Downstream task accuracy dropped only 1.0-1.6 percentage points.

This implies that deep-layer experts are far less input-discriminative than assumed, and that expert capacity is redundant across layers — maintaining a full private expert pool per layer is wasteful.

UniPool's Design

UniPool replaces per-layer private experts with one globally shared expert pool, from which every layer's router selects top-k experts.

Two key innovations:

1. Pool-Level Auxiliary Loss — standard MoE balances load within each layer; UniPool balances utilization across the *entire pool* (L_aux = PoolBalance(f_1, ..., f_M), where f_i is expert i's pool-wide usage frequency), preventing expert collapse when multiple layers compete for the same experts.

2. NormRouter — different layers have different input distributions, causing inconsistent routing-score scales. UniPool applies LayerNorm before computing routing scores, yielding sparse, scale-stable routing and preventing some layers from monopolizing popular experts.

Experimental Results

UniPool was evaluated on five LLaMA-architecture model sizes (182M-978M parameters), trained on 30B tokens of the Pile.

Validation Loss

| Model size | Standard MoE | UniPool | Improvement | |-----------|--------------|---------|-------------| | 182M | 2.847 | 2.808 | -0.039 | | 469M | 2.612 | 2.589 | -0.023 | | 650M | 2.523 | 2.494 | -0.029 | | 830M | 2.461 | 2.422 | -0.039 | | 978M | 2.421 | 2.398 | -0.023 |

Downstream Tasks

Across 7 benchmarks (ARC-Easy, ARC-Challenge, PIQA, HellaSwag, WinoGrande, LAMBADA, RACE), average accuracy improved by roughly +0.8 to +1.9 points, with the largest gain (+1.85) at 830M scale:

| Model size | Standard MoE | UniPool | Gain | |-----------|--------------|---------|------| | 182M | 38.74 | 39.61 | +0.87 | | 469M | 41.62 | 43.11 | +1.49 | | 650M | 43.04 | 43.79 | +0.75 | | 830M | 43.82 | 45.67 | +1.85 | | 978M | 43.91 | 44.07 | +0.16 |

Key Finding: Pool Size as a Depth-Scaling Hyperparameter

In standard MoE, expert parameters grow linearly with depth. UniPool shows sublinear scaling suffices: reduced-pool variants matched or exceeded the standard baseline:

| Configuration | Expert budget | Relative performance | |--------------|---------------|---------------------| | Standard MoE (32 layers x 8 experts) | 100% | Baseline | | UniPool (Pool=16) | 50% | Matches or exceeds | | UniPool (Pool=12) | 37.5% | Close to baseline | | UniPool (Pool=8) | 25% | Still competitive |

Pool size can thus be treated as an explicit scaling hyperparameter, decoupled from layer count.

Fine-Grained Expert Decomposition

Combining the shared pool with finer FFN decomposition into micro-experts brought additional gains at 182M scale:

| Configuration | Avg. accuracy | |--------------|---------------| | Standard MoE (8E/top-1) | 38.74 | | UniPool (8E/top-1) | 39.61 | | UniPool + fine-grained (16 micro-experts) | 40.33 | | UniPool + fine-grained (32 micro-experts) | 41.22 |

Why Sharing Works

• Parameter efficiency: capabilities learned by one layer's expert (e.g., handling math symbols) are reused by all layers instead of being relearned.
• Routing flexibility: each layer chooses from a larger global pool, expanding the per-layer selection space despite fewer total parameters.
• Training stability: pool-level load balancing prevents expert collapse; NormRouter resolves score-scale inconsistency across layers.

Broader Implications

UniPool challenges the default of "privatized" resources per layer. If experts can be shared across layers, the layer's role may be as a processing stage rather than a resource boundary — hinting at future architectures that resemble an expert network rather than a stack of expert layers.

References

1. Huang, M., et al. (2026). UniPool: A Globally Shared Expert Pool for Mixture-of-Experts. *arXiv preprint arXiv:2605.06665*. 2. Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers. *JMLR, 23*(120), 1-39. 3. DeepSeek-AI, et al. (2024). DeepSeekMoE. *arXiv preprint arXiv:2401.06066*. 4. Shazeer, N., et al. (2017). Outrageously Large Neural Networks. *ICLR 2017*. 5. Jacobs, R. A., et al. (1991). Adaptive Mixtures of Local Experts. *Neural Computation, 3*(1), 79-89. 6. Krajewski, J., et al. (2024). Scaling Laws for Fine-Grained Mixture of Experts. *arXiv preprint arXiv:2402.07871*. 7. Zheng, C., et al. (2025). Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel. *arXiv preprint arXiv:2509.25913*.