Mixture-of-Experts (MoE) has become one of the mainstream architectures for large language models. The well-known problem: although each token only activates a small number of experts, the total number of experts keeps growing rapidly. For example, Qwen3.5-397B-A17B has 397B parameters with only 17B activated per token — yet the "cold" experts (the ones not activated) still occupy GPU VRAM.
Processing-in-Memory (PIM) technology — placing compute logic inside memory chips to reduce data movement — has already been implemented in some accelerators. But Sieve (arXiv:2605.11277) points out that existing PIM systems assume a uniform token distribution across experts, while in practice the token-expert distribution of MoE models is bimodal: a few experts receive a large number of tokens, while long-tail experts receive only one or two.
The consequence: compute intensity varies enormously across experts. A static rule such as "compute on GPU, memory-bound work on PIM" is severely inefficient.
Key points
- Problem: Growing MoE parameter counts (e.g., 397B total, 17B active) mean cold experts waste GPU memory; existing PIM offloading assumes uniform expert load, which does not hold in practice.
- Observation: Real token-expert distributions are bimodal — hot experts with heavy token traffic coexist with long-tail experts handling one or two tokens, so expert compute intensity differs drastically.
- Approach — Sieve: A dynamic scheduler decides at runtime, per expert, whether execution goes to the GPU or to PIM, based on live factors including the token-expert distribution, network overhead, and available bandwidth.
- Results: On models such as Qwen3.5-397B, Sieve improves throughput and interactivity by 1.3–1.6x.
- Caveat: Evaluation was done on the Ramulator 2.0 cycle-accurate simulator; real-hardware performance may deviate from simulation results.
References
1. Kim, J., et al. (2026). *Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Models*. arXiv:2605.11277 [cs.AR]. 2. Shazeer, N., et al. (2017). *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. ICLR 2017. 3. Lee, S., et al. (2024). *HBM-PIM: Processing-in-Memory for High-Bandwidth Memory*.