Running a 35B MoE Model on a Laptop: The Memory-Swapping Magic Explained

Forum topic · 小凯 · 2026-04-19

Summary

This article explains how a 35-billion-parameter mixture-of-experts (MoE) language model, Qwen3.5-35B-A3B, can run locally on a laptop with an RTX 5080 (16GB VRAM). It contrasts dense models, where all parameters compute every token, with MoE architectures that activate only 8 of 128 experts per token, cutting effective compute to roughly 3B parameters. The catch: all expert weights must still reside in memory, totaling about 90GB with FP16 weights and 128K-context KV cache. The solution combines quantization (Q4/Q6 GGUF shrinking weights to 17.5GB or less) with LM Studio's setting to force MoE weights of certain layers onto CPU RAM, loading experts on demand. Reported speeds are ~30 tokens/s at Q6 and ~45 tokens/s at Q4, with 128K-token context and intact vision capabilities. The piece also covers why local deployment matters (privacy, offline use, low latency), MoE's trade-offs (fine-tuning cost, load balancing, communication overhead), and a practical three-step setup guide.

Introduction

Imagine running a 35-billion-parameter large language model on a laptop with an RTX 5080 (16GB VRAM) — at over 30 tokens/s quantized to Q6, or 45+ tokens/s at Q4, with 128K-token context and intact vision capabilities. This is achievable thanks to the Mixture of Experts (MoE) architecture plus a "RAM-for-VRAM" optimization trick.

Why dense models don't fit

Traditional dense models activate all parameters for every token. A 35B model in FP16 needs:

35 \text{B} \times 2 \text{ bytes} = 70 \text{GB}

Far beyond any consumer GPU.

How MoE works

Expert networks: Qwen3.5-35B-A3B has 128 parallel expert FFNs, each specialized for certain input types.
Gating network: for each token, it scores all experts and activates only the top 8 of 128.
Sparse activation: effective compute is roughly 35B × 8/128 ≈ 2.2B (A3B = activated 3B) parameters per token — but the *full* weight set must still be loaded.

Memory budget for Qwen3.5-35B-A3B (FP16, 128K context):

| Component | VRAM/RAM | |------|----------| | Model weights (FP16) | ~70 GB | | KV cache (128K) | ~16 GB | | Activations | ~4 GB | | Total | ~90 GB |

The trick: quantization + MoE CPU offload

1. Quantization: Q6 stores each parameter in 6 bits; Q4 in 4 bits, shrinking 35B weights to ~17.5GB (Q4). 2. MoE layer offload: LM Studio's setting — "Number of layers for which to force MoE weights onto CPU" — parks expert weights for the first 20–35 layers in system RAM. Since most experts are idle for any given token, they can be loaded on demand and cached, so speed recovers after warm-up. This only works for MoE models; dense models would slow to a crawl from constant copying.

Measured results on an RTX 5080 laptop:

| Configuration | Speed | VRAM | |------|----------|----------| | Q6 + MoE offload | ~30 token/s | ~14–15 GB | | Q4 + MoE offload | ~45 token/s | ~10–12 GB | | Dense model (same scale) | Cannot run | >16 GB |

Why local deployment matters

Privacy: data never leaves the device.
Offline availability: works on planes, remote areas, restricted networks.
Low latency: better for coding assistants and real-time writing help.
Multimodal: the uncensored variant retains vision — photo tagging, chart analysis.
128K context: summarize whole books, analyze long transcripts, cross-file code refactoring.

MoE's limits

Fine-tuning requires all expert weights (for backpropagation), scaling with total parameters, not active ones.
Load balancing: the gate can over-favor certain experts ("Matthew effect"), needing special training techniques.
All-to-All communication overhead in multi-GPU distributed training.

Practical setup guide

1. Download the Qwen3.5-35B-A3B GGUF (Q4 or Q6) from HuggingFace or ModelScope. 2. In LM Studio: set GPU Offload to max; set "Number of layers for which to force MoE weights onto CPU" to ~20–35 (start at 25 and tune). 3. Chat — the first response may be slow while experts load; it speeds up afterward.

Closing thought

MoE + smart memory management represents the democratization of AI: a model that once needed datacenter hardware now runs smoothly on a laptop, enabling privacy, offline use, and local multimodal processing for everyone.

References

1. Qwen3.5 Technical Report — Alibaba Qwen team, Feb 2026. 2. "MoE高效训练的A/B面" — AI科技评论, May 2024. 3. Shazeer et al., 2017 — *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. 4. LM Studio Documentation — lmstudio.ai. 5. Muennighoff et al., 2025 — *Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient*.