PagedWeight: Dynamic Quality-Aware Weight Quantization for Efficient MoE LLM Serving

Forum topic · 小凯 · 2026-07-20

Summary

This post from zhichai.net reviews the paper 'PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization' (arXiv:2607.16184). The paper addresses GPU memory contention in Mixture-of-Experts (MoE) large language model inference, where model weights, KV cache, and activations must coexist in limited VRAM. Unlike static quantization applied uniformly after training, PagedWeight treats expert weights as paged memory inspired by operating systems: each expert's weight matrices are split into fixed-size pages, tracked in a page table, and managed with LRU-style swapping between GPU, CPU, and compressed states. A quality-aware mechanism maintains an Expert Activity Table that scores expert importance at runtime, keeping hot experts at FP16, warm experts at INT8, and cold experts at INT4, adjusting precision progressively based on current memory pressure. Experiments on Mixtral 8x7B and 8x22B against baselines like AWQ, GPTQ, and static INT8/INT4 show up to 72% GPU memory savings at FP16-equivalent quality, 1.94x throughput gains in memory-bound scenarios, up to 39.3% accuracy advantage under tight budgets, and at most 4.1% throughput overhead. The post also discusses future directions including speculative decoding synergy and joint KV-cache compression.

This article from zhichai.net is a plain-language explainer of the paper PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization (arXiv: 2607.16184, published 2026-07-17, authors Yuchen Yang, Yifan Zhao, Anisha Dasgupta, et al.).

The Problem: GPU Memory as a Tiny Apartment

During LLM inference, GPU memory must simultaneously hold:

Model weights — a 70B-parameter model needs ~140GB at FP16
KV cache — the Transformer's short-term memory, growing with context length (tens of GB for 32K contexts)
Activations — intermediate forward-pass results

For Mixture-of-Experts (MoE) models (e.g., DeepSeek-V3, Qwen2.5-MoE, Mixtral), only a few experts are activated per token, but all experts' weights must reside in GPU memory since the router's future choices are unpredictable.

Key points

Core insight: Not all experts are equally important. The router activates some experts frequently (code, math) and others rarely — so quantization precision should vary by expert importance.
Quality-aware dynamic quantization: PagedWeight maintains an Expert Activity Table recording recent activation frequency, updated dynamically with input distribution. Experts are tiered as:
Hot experts → FP16
Warm experts → INT8
Cold experts → INT4 or lower
Paged memory management borrowed from OS design: each expert's weight matrices are split into fixed-size pages (e.g., 64MB), tracked in a page table recording location (GPU/CPU/compressed), precision level, and last-access time. Pages are swapped under an LRU policy, and only pages needed by the current batch are loaded. Different pages of the same expert can even carry different precisions.
Continuous, progressive adjustment: rather than a three-tier switch, PagedWeight optimizes a quality–memory tradeoff function. Under memory pressure it compresses experts by maximizing the ratio of memory freed to quality lost; adjustments are gradual (threshold-triggered) rather than reactive, keeping overhead low.

Experimental results

Evaluated on Mixtral 8x7B and Mixtral 8x22B against static INT8/INT4 quantization, AWQ, and GPTQ, across long-conversation, high-concurrency, and mixed workloads:

72.0% GPU memory savings at FP16-equivalent quality — e.g., a model needing 80GB fits in ~22.4GB, enabling consumer GPUs (RTX 4090, 24GB) to run models that previously required an A100-80GB.
1.94x throughput improvement in memory-bound scenarios, by freeing space for the KV cache and reducing GPU–CPU data movement.
39.3% quality advantage (downstream task accuracy) under strict memory budgets compared to static quantization.
At most 4.1% throughput loss, thanks to progressive, predictive rather than frequent reactive adjustments.

Takeaways and future directions

PagedWeight marks a shift from static, one-shot optimization toward runtime-adaptive inference systems. Suggested future work includes:

Combining freed memory with speculative decoding draft models
Extending dynamic quantization to other partially-activated architectures (sparse Transformers, conditional computation)
Joint optimization with KV-cache compression methods (H2O, StreamingLLM)
Hardware-aware tuning of page size and compression granularity

References cited in the post

1. Yang, Y., et al. (2026). PagedWeight. arXiv:2607.16184 2. Fedus, Zoph & Shazeer (2022). Switch Transformers. JMLR 23(120) 3. Dettmers et al. (2022). LLM.int8(). NeurIPS 2022 4. Frantar et al. (2023). GPTQ. ICLR 2023 5. Lin et al. (2024). AWQ. MLSys 2024 6. Jiang et al. (2024). Mixtral of Experts. arXiv:2401.04088 7. Liu et al. (2023). Deja Vu. ICML 2023 8. Zhang et al. (2024). H2O. NeurIPS 2023

*Explainer: Xiaokai | Feynman-style deep dive | 2026-07-21*