Mixture-of-Experts (MoE) has become one of the mainstream architectures for LLMs. The well-known problem: although each token only activates a few experts, the total number of experts keeps growing. Qwen3.5-397B-A17B has 397B parameters but activates only 17B per token — yet the cold experts (the ones not activated) still occupy GPU memory.
Processing-in-Memory (PIM) technology — putting compute logic inside memory chips to reduce data movement — has already been implemented in some accelerators. But Sieve (arXiv:2605.11277) points out that existing PIM systems assume a uniform token distribution across MoE experts, while in reality the token-expert distribution is bimodal: a small number of experts receive a large number of tokens, while long-tail experts receive only one or two.
The result: huge differences in compute intensity across experts. A unified static rule like "compute goes to GPU, memory-bound work goes to PIM" is severely inefficient.
Sieve proposes a dynamic scheduler: based on runtime factors such as the token-expert distribution, network overhead, and bandwidth, it dynamically decides whether each expert's execution should run on the GPU or on PIM. On models including Qwen3.5-397B, both throughput and interactivity improve by 1.3-1.6x.
Caveat: Sieve's evaluation is based on the Ramulator 2.0 cycle-accurate simulator — real-hardware performance gaps may differ from simulation results.
References
1. Kim, J., et al. (2026). *Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Models*. arXiv:2605.11277 [cs.AR]. 2. Shazeer, N., et al. (2017). *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. ICLR 2017. 3. Lee, S., et al. (2024). *HBM-PIM: Processing-in-Memory for High-Bandwidth Memory*.