Meta's MobileMoE: Putting a Mixture-of-Experts LLM on Your Phone

Forum topic · 小凯 · 2026-06-02

Summary

A zhichai.net forum post analyzes MobileMoE (arXiv 2605.27358), a Meta AI paper introducing the first on-device MoE scaling law under joint memory and compute constraints. The paper finds an architectural sweet spot—moderate sparsity, fine-grained experts, and shared experts—derived via constrained optimization (optimal expert count: 8 at a 5GB memory budget with INT4 quantization). The team trained a family of models (0.3B/0.5B/0.9B active, 1.3B/2.8B/5.3B total parameters) through a four-stage pipeline: pre-training on ~6T tokens, mid-training, instruction fine-tuning, and quantization-aware training. On 14 benchmarks, MobileMoE sets a new on-device Pareto frontier: 2-4x fewer inference FLOPs than dense baselines with comparable memory, and roughly 60% fewer parameters than OLMoE-1B-7B at equal accuracy. On commercial smartphones with INT4 weights, prefill is 1.8-3.8x faster and decode 2.2-3.4x faster than the dense MobileLLM-Pro baseline. The post also raises open questions: routing overhead, cold-start load time, multi-app memory contention, scaling-law generalization beyond the fitted range, and interaction with on-device RLVR post-training.

This post is a Chinese-language forum review of MobileMoE: Scaling On-Device Mixture of Experts (Meta AI; arXiv 2605.27358, submitted May 26, 2026). Below is a structured English translation/summary.

Key points

The on-device compute dilemma

Cloud LLMs run on data-center GPUs; on-device AI is constrained by DRAM (typically 4-8GB shared with the OS and apps), battery-limited compute, and latency.
MoE decouples total parameters (memory) from active parameters (compute), offering an asymmetric trade: more memory for less computation. But cloud MoE design principles do not transfer directly to phones, where memory and FLOPs are two independent hard constraints.

A new on-device MoE scaling law

The paper formulates a scaling law with explicit constraints on compute (training/inference FLOPs) and memory (INT4 weights + KV cache within a ~5GB budget), optimized via nonlinear least squares (L-BFGS-B).
Under these joint constraints, the sweet spot is moderate sparsity + fine-grained experts + shared experts. The optimal expert count solves to 8 — not 32, not 2.
The formulation degrades gracefully: fixing architecture recovers prior joint MoE scaling laws; fixing expert count recovers Chinchilla-style scaling.

Four-stage training pipeline

Pre-training

Mid-training

Instruction fine-tuning

Quantization-aware training (QAT)

A shared expert (always active) provides a stable capability floor and guards against catastrophic routing decisions.

Results

Model family: MobileMoE-S (0.3B active / 1.3B total), -M (0.5B / 2.8B), -L (0.9B / 5.3B); INT4 weights fit under 3GB.
New Pareto frontier across 14 benchmarks (commonsense, knowledge, science, understanding, reasoning).
vs. dense baselines: matching or better accuracy with 2-4x fewer inference FLOPs at comparable memory.
vs. OLMoE-1B-7B: MobileMoE-M matches accuracy with ~60% fewer active and total parameters; MobileMoE-L is more accurate with 30% fewer active params and 23% less memory.
On commercial smartphones (INT4 weights, 8-bit KV cache): prefill 1.8-3.8x faster, decode 2.2-3.4x faster than dense MobileLLM-Pro.

Alternatives considered

More aggressive quantization (INT2/INT3): steep accuracy loss beyond INT4+QAT.
Speculative decoding: needs two models, worsening memory.
Distillation: student capped by teacher.
Dynamic depth: saves layers vertically vs. MoE's horizontal expert sparsity; combining both is an open direction.

Open questions raised by the author

Routing overhead and load-balancing quality on phone GPUs (conditional routing causes scattered memory access).
Cold-start load time for ~2.6GB of weights from flash.
Multi-app memory contention on device.
Generalization of the scaling law beyond the fitted range (0.3-0.9B active, 100-500B tokens, 1-32 experts) and to other data/hardware.
Interaction with on-device RLVR post-training and potential expert specialization.

Paper metadata

| Item | Detail | |------|--------| | Title | MobileMoE: Scaling On-Device Mixture of Experts | | Authors | Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi | | Institution | Meta AI | | arXiv ID | 2605.27358 | | Categories | cs.CL, cs.LG, cs.AI | | Core method | On-device MoE scaling law; moderate sparsity + fine-grained + shared experts; four-stage training with QAT | | Headline results | 2-4x fewer FLOPs vs. dense; 60% fewer params vs. OLMoE-1B-7B; 1.8-3.8x prefill / 2.2-3.4x decode speedup on smartphones |