MobileMoE: Meta Brings Mixture-of-Experts Scaling Laws to On-Device LLMs

Forum topic · 小凯 · 2026-06-02

Summary

A detailed Chinese-language forum analysis of MobileMoE, a Meta AI paper (arXiv 2605.27358) introducing the first on-device Mixture-of-Experts (MoE) scaling law under joint memory and compute constraints. The post explains how the optimal architecture for smartphones differs from cloud MoE design: moderate sparsity, fine-grained experts, plus shared experts, with an optimal expert count of 8 under a 5GB DRAM budget and INT4 quantization. The MobileMoE family (0.3B/0.5B/0.9B active parameters, 1.3B/2.8B/5.3B total) is trained via a four-stage pipeline ending in quantization-aware training. Results include 2-4x fewer inference FLOPs than dense baselines, roughly 60% fewer parameters than OLMoE-1B-7B at matched accuracy, and real smartphone speedups of 1.8-3.8x prefill and 2.2-3.4x decode versus MobileLLM-Pro. The author also raises open questions about routing overhead, cold-start latency, and scaling law generalization.

Pocket-Sized Expert Committee: Meta Squeezes Billion-Parameter MoE Models onto Phones

This post is a translation/summary of a Chinese forum analysis of the Meta AI paper MobileMoE: Scaling On-Device Mixture of Experts (arXiv 2605.27358, submitted May 26, 2026, cs.CL/cs.LG/cs.AI), by Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, and Raghuraman Krishnamoorthi.

Key points

On-device MoE scaling law: The authors propose a scaling law that jointly optimizes under two hard constraints — FLOPs budget and memory budget (INT4 weights plus KV cache, ~5GB DRAM). The derived optimum is moderate sparsity + fine-grained experts + shared experts, with an optimal expert count of 8 at 0.3B/0.5B/0.9B active scales. The formula degrades to known joint MoE scaling laws and Chinchilla scaling laws in limiting cases.
Cloud intuition doesn't transfer: Cloud MoE favors many experts and high sparsity; on phones, over-sparsity inflates total parameters and blows the memory budget, while coarse-grained experts waste MoE's compute savings.
Four-stage training pipeline: Pre-training (~6T tokens, fewer than dense baselines like Llama 3.2 1B's 9T), mid-training on dense data, instruction fine-tuning, and quantization-aware training (QAT) for INT4 deployment. A shared expert is always active to stabilize routing.
Model family: MobileMoE-S (0.3B active / 1.3B total), M (0.5B / 2.8B), L (0.9B / 5.3B). INT4 weights stay under 3GB.

Benchmark and deployment results

New Pareto frontier across 14 foundational benchmarks (commonsense, knowledge, science, understanding, reasoning).
vs. dense baselines: 2–4x fewer inference FLOPs at matched or better accuracy.
vs. OLMoE-1B-7B: MobileMoE-M achieves the same accuracy with ~60% fewer active and total parameters; MobileMoE-L is more accurate with 30% fewer active params and 23% less memory.
On commodity smartphones (INT4 weights, 8-bit KV cache): prefill 1.8–3.8x faster and decode 2.2–3.4x faster than dense MobileLLM-Pro — first demonstration of efficient MoE inference on a commercial phone.

Why MoE fits phones

On-device AI faces two independent constraints: DRAM capacity (how large a model you can store) and FLOPs (how fast and energy-efficient inference is). MoE decouples total parameters (memory) from active parameters (compute), trading extra memory for less computation. The challenge is MoE's complexity — routing overhead, expert load imbalance, and training instability — which cloud-scale engineering absorbs easily but small on-device training budgets do not.

Author's open questions

The post honestly flags unresolved issues:

1. Routing overhead and load balance on phone GPUs (MoE causes random memory access vs. dense models' contiguous access). 2. Cold-start latency when loading multi-GB weights from flash. 3. Coexistence with other memory-hungry AI apps. 4. Generalization of the scaling law across datasets, tokenizers, and hardware (e.g., RISC-V). 5. Interaction with on-device RLVR post-training — whether experts would functionally specialize for reasoning vs. commonsense.

The author also considers alternatives (aggressive INT2/INT3 quantization, speculative decoding, knowledge distillation, dynamic depth) and argues MoE's architectural-level optimization is the most balanced path under strict memory and latency constraints.

> Original post tags: #CrushAI #FeynmanLearning #智柴系统实验室🎙️