Mixture-of-Experts (MoE) is now a mainstream LLM architecture, but it has a well-known problem: although each token only activates a few experts, the total number of experts keeps growing rapidly. For example, Qwen3.5-397B-A17B has 397B parameters while activating only 17B per token — yet the "cold" (inactive) experts still occupy GPU memory.
Processing-in-Memory (PIM), which moves compute logic into memory chips to reduce data movement, has already been adopted in some accelerators. But the paper Sieve (arXiv:2605.11277) points out that existing PIM systems assume MoE token distributions are uniform, when in reality the token-to-expert distribution is bimodal: a small number of experts receive large numbers of tokens, while long-tail experts receive only one or two.
The consequence: per-expert compute intensity varies enormously. A static rule such as "compute goes to GPU, memory-bound work goes to PIM" is severely inefficient.
Sieve proposes a dynamic scheduler: at runtime, it decides whether each expert's execution goes to the GPU or PIM based on the live token-expert distribution, network overhead, bandwidth, and other real-time factors. On models like Qwen3.5-397B, throughput and interactivity improve by 1.3–1.6x.
Caveat
Sieve's evaluation is based on the Ramulator 2.0 cycle-accurate simulator — the performance gap on real hardware may differ from the simulation results.
References
1. Kim, J., et al. (2026). *Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Models*. arXiv:2605.11277 [cs.AR]. 2. Shazeer, N., et al. (2017). *Outrageously Large Neural Networks: Sparsely-Gated Mixture-of-Experts Layer*. ICLR 2017. 3. Lee, S., et al. (2024). *HBM-PIM: Processing-in-Memory for High-Bandwidth Memory*.