Running a 35B MoE LLM on a Laptop: How Mixture-of-Experts Enables Local Inference on RTX 5080

Forum topic · 小凯 · 2026-04-19

Summary

This article explains how a 35-billion-parameter language model, specifically Qwen3.5-35B-A3B, can run locally on a consumer laptop such as an RTX 5080 (16 GB VRAM). The key is the Mixture-of-Experts (MoE) architecture, which holds 128 experts but activates only about 3 billion parameters per token, drastically reducing compute. Combined with Q4/Q6 quantization and an LM Studio setting that forces a configurable number of MoE layers onto CPU memory, the model achieves roughly 30 tokens/s at Q6 and 45 tokens/s at Q4, while supporting 128K context and retaining vision capabilities. The piece covers MoE core mechanics, the hidden memory cost of loading all experts, the CPU-VRAM offload trade-off, practical setup steps, and broader implications for on-device AI, including privacy, offline use, and low latency.

A Seemingly Impossible Feat

Imagine sitting in a café, opening your laptop, pasting a 100,000-character document, and asking the machine to summarize its core arguments and find connections between data points. Seconds later, you receive a clear, accurate analysis. Until recently, that level of long-context understanding required server-class GPUs. Yet on an RTX 5080 laptop GPU with 16 GB of VRAM, a 35-billion-parameter LLM (Qwen3.5-35B-A3B) now runs locally at over 30 tokens/s in Q6 quantization and 45+ tokens/s in Q4, with a 128K-token context window and vision capabilities intact. The trick is the Mixture-of-Experts architecture combined with a CPU-memory offload technique.

Chapter 1: The "All-Hands" Problem of Dense Models

Traditional dense Transformers (GPT, Llama dense variants) activate every parameter for every token. A 35B model at FP16 requires roughly:

$$35 \text{B} \times 2 \text{ bytes} = 70 \text{ GB}$$

That exceeds even an RTX 4090's 24 GB, leaving consumers limited to 7B-13B models that feel underpowered for complex reasoning, long documents, and multilingual tasks.

Chapter 2: Mixture-of-Experts — Specialization by Design

MoE replaces the single feed-forward block with many parallel expert networks, plus a gating network that routes each token to a small subset of experts. Qwen3.5-35B-A3B has 128 experts but activates only 8 per token, yielding roughly 2.2B activated parameters (officially labeled A3B, ~3B).

Per-token compute drops dramatically:

$$35 \text{B} \times \frac{8}{128} \approx 2.2 \text{ B}$$

Inference VRAM for active weights falls to around 6 GB. Yet total capacity remains that of a 35B model.

Chapter 3: The Hidden Cost — All Experts Must Be Loaded

Sparse activation lowers compute, not storage. The gating network can pick *any* expert at any step, so the runtime must keep all experts available. Total footprint for Qwen3.5-35B-A3B at FP16 with 128K context is roughly:

| Component | VRAM | |---|---| | Weights (FP16) | ~70 GB | | KV cache (128K) | ~16 GB | | Activations & scratch | ~4 GB | | Total | ~90 GB |

This is the classic "VRAM-for-performance" trade-off of MoE.

Chapter 4: Quantization + CPU Offload — The Memory Magic

Quantization shrinks each weight:

Q6 → ~26 GB model
Q4 → ~17.5 GB model

Still too large for 16 GB VRAM. The MoE-specific trick: since only 8 of 128 experts run at any moment, the others can sit in CPU RAM. LM Studio exposes this via the setting "Number of layers for which to force MoE weights onto CPU", typically set to 20–35.

Workflow:

1. Initialize: front layers' experts on GPU, the rest in CPU RAM. 2. Infer: GPU pulls the needed experts layer by layer. 3. Dynamic load: cold experts copy from RAM with a small first-call delay. 4. Warm-up: hot experts stay cached in VRAM, so steady-state speed approaches full VRAM deployment.

Measured on RTX 5080 (16 GB):

| Config | Speed | VRAM | |---|---|---| | Q6 + MoE layer split | ~30 tok/s | ~14–15 GB | | Q4 + MoE layer split | ~45 tok/s | ~10–12 GB | | Q6 dense (same scale) | won't fit | >16 GB |

The offload only works for MoE; dense models would stall from constant CPU↔GPU copies.

Chapter 5: Why Local AI Matters

Privacy & sovereignty: data never leaves the device — critical for journals, medical records, IP.
Offline reliability: usable on planes, in remote areas, behind firewalls.
Low latency: faster turnarounds than cloud APIs for coding and writing assistants.
Multimodality: vision variants tag images, parse charts, and analyze video timelines locally.
Long context: 128K fits an entire ~300-page book, multi-hour transcripts, or whole codebases.

Chapter 6: Is MoE the Future?

Most frontier models are now MoE:

| Model | Total | Active | Vendor | |---|---|---|---| | Mixtral 8x7B | 46.7B | ~13B | Mistral AI | | DeepSeek-V2 | 236B | 21B | DeepSeek | | Qwen3-235B-A22B | 235B | 22B | Alibaba | | Qwen3.5-35B-A3B | 35B | 3B | Alibaba | | GLM-4.5 | 32B | ~8B | Zhipu |

Open challenges:

Fine-tuning loads all experts for backprop, so personal fine-tuning is VRAM-bound by total size, not active size.
Load balancing between experts requires auxiliary losses to avoid routing collapse.
All-to-all communication between GPUs is a training bottleneck.

Setup Guide (Three Steps)

1. Get the model: download a Q4 or Q6 GGUF of Qwen3.5-35B-A3B from HuggingFace or ModelScope. 2. Configure LM Studio: maximize GPU Offload for your VRAM; set *Number of layers for which to force MoE weights onto CPU* to 20–35 (try 25 first). 3. Chat: first response may be slower as experts warm up; subsequent turns accelerate.

Key Points

MoE keeps total knowledge high while activating only ~3B of 35B parameters per token.
All 35B parameters must still be loadable, so VRAM is the binding constraint.
Quantization (Q4/Q6) plus LM Studio's MoE CPU-offload layer setting brings a 35B model onto a 16 GB laptop GPU.
Practical speeds: ~30 tok/s (Q6) and ~45 tok/s (Q4) with 128K context on RTX 5080.
Trade-off is first-token latency for cold experts, mitigated by caching of hot experts.
This pattern enables private, offline, low-latency local AI — a meaningful step in democratizing frontier models.

References

1. Qwen3.5 Technical Report — Alibaba Tongyi Team, February 2026. 2. "MoE高效训练的A/B面：与魔鬼做交易，用显存换性能" — AI Tech Review, May 2024. 3. Shazeer et al., 2017, *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*. 4. LM Studio Documentation — lmstudio.ai. 5. Muennighoff et al., 2025, *Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient*.