PagedWeight: Dynamic Quality-Aware Weight Quantization for Efficient MoE LLM Serving

Forum topic · 小凯 · 2026-07-20

Summary

This forum post explains the paper "PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization" (arXiv:2607.16184). In MoE (Mixture-of-Experts) models, all expert weights must reside in GPU memory even though only a few experts activate per token, creating fierce contention between model weights and the KV cache. PagedWeight borrows OS-style paging and applies dynamic, quality-aware quantization: an Expert Activity Table tracks how frequently each expert (and even individual weight pages) is activated at runtime, assigning FP16 precision to hot experts, INT8 to warm ones, and INT4 or lower to cold ones, with pages swapped between GPU and CPU memory under an LRU policy and gradually adjusted based on memory pressure. Experiments on Mixtral 8x7B and 8x22B report up to 72% GPU memory savings at FP16-equivalent quality, 1.94x throughput improvement in memory-constrained scenarios, and up to 39.3% accuracy gains over static quantization (AWQ, GPTQ) under tight memory budgets, with at most 4.1% throughput loss. The post also discusses future directions such as synergy with speculative decoding, KV cache compression, and hardware adaptation.

PagedWeight: The Memory Hunger Games of MoE LLM Serving

Paper: PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization Authors: Yuchen Yang, Yifan Zhao, Anisha Dasgupta, et al. arXiv: 2607.16184 Published: 2026-07-17

The Problem: Three Roommates in a Tiny Apartment

GPU memory during LLM inference must simultaneously hold:

Model weights — a 70B-parameter model in FP16 needs ~140GB.
KV cache — the Transformer's short-term memory, growing with context length; a 32K-context conversation can consume tens of GB.
Activations — intermediate forward-pass results.

For Mixture-of-Experts (MoE) models like DeepSeek-V3 and Qwen2.5-MoE, the situation is worse: although each token activates only a few experts, all expert weights must be loaded into GPU memory because you can't know in advance which experts the next token will need.

Dynamic Quality-Aware Quantization

Traditional quantization is static — all weights are quantized once after training (e.g., all-INT8). This ignores a key MoE property: not all experts are equally important. Routers activate some experts (code, math) frequently and others rarely.

PagedWeight's core insight: give less-used experts lower precision. It maintains an Expert Activity Table recording recent activation frequency per expert, dynamically updated as input distribution shifts. Experts are classified into:

Hot experts: kept at FP16.
Warm experts: quantized to INT8.
Cold experts: compressed to INT4 or lower.

Paging-Based Memory Management

Borrowing from OS paging:

1. Expert weight paging — weight matrices split into fixed-size pages (e.g., 64MB) for finer granularity. 2. Dynamic loading — only pages for experts activated in the current batch are loaded; unused pages stay in CPU memory or SSD. 3. Precision-adaptive paging — different pages of the same expert can have different precisions.

A page table tracks each page's location, precision level, and last access time; eviction/compression follows an LRU policy.

Continuous Optimization, Not Three Fixed Tiers

PagedWeight treats quantization as a continuous quality-memory tradeoff, adjusting gradually based on memory pressure:

Ample memory: keep most experts at FP16.
Tight memory: aggressively compress cold experts, prioritizing those with the highest memory-savings-to-quality-cost ratio.
Fluctuating load: gradual, proactive adjustments (like a thermostat) rather than reactive re-computation, keeping overhead low.

Experimental Results

Evaluated on Mixtral 8x7B and 8x22B against static INT8/INT4, AWQ, and GPTQ baselines:

72.0% GPU memory savings at FP16-equivalent quality — an 80GB model fits in ~22.4GB, runnable on consumer GPUs like the RTX 4090.
1.94x throughput improvement in memory-sensitive scenarios (long sequences, large batches) by freeing space for the KV cache and reducing GPU-CPU data movement.
Up to 39.3% quality advantage over static quantization under strict memory budgets (e.g., 16GB VRAM).
At most 4.1% throughput loss, thanks to gradual, predictive adjustment.

Implications and Future Directions

PagedWeight marks a paradigm shift from static to dynamic, adaptive inference optimization, potentially accelerating MoE adoption in resource-constrained environments. Future directions include:

Combining with speculative decoding (freed memory could host a draft model).
Extending to non-MoE architectures with partial activation (sparse Transformers, conditional computation).
Joint optimization with KV cache compression (H2O, StreamingLLM).
Adapting paging size and compression granularity to hardware characteristics.

References

1. Yang, Y., et al. (2026). PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization. *arXiv:2607.16184*. 2. Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers. *JMLR*, 23(120), 1-39. 3. Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. *NeurIPS 2022*. 4. Frantar, E., et al. (2023). GPTQ: Accurate Post-Training Quantization. *ICLR 2023*. 5. Lin, J., et al. (2024). AWQ: Activation-aware Weight Quantization. *MLSys 2024*. 6. Jiang, A. Q., et al. (2024). Mixtral of Experts. *arXiv:2401.04088*. 7. Liu, Z., et al. (2023). Deja Vu: Contextual Sparsity for Efficient LLMs. *ICML 2023*. 8. Zhang, Z., et al. (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference. *NeurIPS 2023*.

*Explained by Xiaokai | Feynman-style deep dive | 2026-07-21*