MoE Models Keep Growing but GPU Memory Can't Keep Up — Can Processing-in-Memory Chips Help?

Forum topic · 小凯 · 2026-05-18

Summary

Mixture-of-Experts (MoE) models are now a mainstream LLM architecture, but total expert counts are growing fast: for example, Qwen3.5-397B-A17B holds 397B parameters while activating only 17B per token, meaning inactive (cold) experts still occupy scarce GPU memory. A paper called Sieve (arXiv:2605.11277) argues that existing Processing-in-Memory (PIM) systems, which place compute logic inside memory chips to reduce data movement, wrongly assume a uniform token-to-expert distribution. In practice the distribution is bimodal: a few experts receive large token volumes while long-tail experts receive only one or two, creating large per-expert compute intensity differences. A static rule like 'compute on GPU, memory-bound work on PIM' is therefore inefficient. Sieve proposes a dynamic scheduler that assigns each expert's execution to GPU or PIM at runtime based on the token-expert distribution, network overhead, and bandwidth. Simulated on models including Qwen3.5-397B using Ramulator 2.0, it reports 1.3-1.6x throughput and interactivity gains, though real-hardware results may differ from cycle-accurate simulation.

Mixture-of-Experts (MoE) has become one of the mainstream architectures for LLMs. The well-known problem: although each token only activates a few experts, the total number of experts keeps growing. Qwen3.5-397B-A17B has 397B parameters but activates only 17B per token — yet the cold experts (the ones not activated) still occupy GPU memory.

Processing-in-Memory (PIM) technology — putting compute logic inside memory chips to reduce data movement — has already been implemented in some accelerators. But Sieve (arXiv:2605.11277) points out that existing PIM systems assume a uniform token distribution across MoE experts, while in reality the token-expert distribution is bimodal: a small number of experts receive a large number of tokens, while long-tail experts receive only one or two.

The result: huge differences in compute intensity across experts. A unified static rule like "compute goes to GPU, memory-bound work goes to PIM" is severely inefficient.

Sieve proposes a dynamic scheduler: based on runtime factors such as the token-expert distribution, network overhead, and bandwidth, it dynamically decides whether each expert's execution should run on the GPU or on PIM. On models including Qwen3.5-397B, both throughput and interactivity improve by 1.3-1.6x.

Caveat: Sieve's evaluation is based on the Ramulator 2.0 cycle-accurate simulator — real-hardware performance gaps may differ from simulation results.

References

1. Kim, J., et al. (2026). *Sieve: Dynamic Expert-Aware PIM Acceleration for Evolving Mixture-of-Experts Models*. arXiv:2605.11277 [cs.AR]. 2. Shazeer, N., et al. (2017). *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. ICLR 2017. 3. Lee, S., et al. (2024). *HBM-PIM: Processing-in-Memory for High-Bandwidth Memory*.

MoE Models Keep Growing but GPU Memory Can't Keep Up — Can Processing-in-Memory Chips Help?

Summary

References

Tags