Easy AI Tutorial: MoE (Mixture of Experts) Architecture
> This is an English translation of an Easy AI tutorial originally published on zhichai.net about the Mixture of Experts (MoE) model architecture.
One-Sentence Definition
MoE is an architecture that lets an AI model have a massive total parameter count while keeping inference extremely fast. It decouples a model's *total parameters* from its *activated parameters* — the key difference from traditional dense models.
Core Metaphor: A General Hospital
- Dense model = an all-in-one doctor who masters every department. Every patient (token), simple or complex, requires mobilizing all the doctor's knowledge — heavy memory burden, slow and exhausting.
- MoE = a team of specialists (cardiology, neurology, dermatology...). A reception desk (the Router) assigns each patient to only the most relevant 1–2 experts for consultation; the others rest.
- Lower training cost: train a larger-capacity model with the same compute, breaking through the Scaling Law bottleneck.
- Extremely fast inference: a 100B-parameter model only activates ~10B per token — response speed like lightning.
- Long-context support: high computational efficiency makes MoE easier to scale to ultra-long context windows.
- Huge VRAM demands: although only a small fraction of parameters is used per computation, all parameters must be loaded into VRAM (or swapped rapidly in memory). VRAM requirements far exceed those of a dense model with equal compute — very unfriendly to consumer GPUs.
- Training load balancing: early in training, the router can develop preferences, routing all tasks to one or a few experts — causing those experts to overwork while others idle (collapse problem). Complex auxiliary losses are needed so all experts get trained ("rain falls evenly on all").
- GPT-4 (industry benchmark): widely believed to be a huge MoE model (rumored: 8 experts of ~220B parameters each).
- Mixtral 8x7B (open-source milestone): 8 experts, 2 activated per token; 47B total parameters with inference compute equivalent to a 13B model, yet outperforming LLaMA-2 70B.
- DeepSeek-V2 / V3 (DeepSeek): innovative MLA (Multi-head Latent Attention) combined with MoE, drastically reducing VRAM usage and inference cost — a frontier of MoE optimization.
- Grok-1 (xAI): Elon Musk's open-sourced 314B-parameter model, also a typical MoE.
Technical Mechanism
In a Transformer, MoE replaces the original FFN (feed-forward network) layer with two components:
1. Expert networks: a set of parallel feed-forward neural networks that automatically differentiate during training, each specializing in domains like grammar, code, or logic. 2. Gating network (Router): computes weights (probabilities) deciding which experts a token should go to. For each token, only the top-probability experts are activated; if an expert's weight is 0, it does not participate in the computation.
The core formula combines the gating network's weights with the outputs of the selected expert networks.
Key Advantages
Challenges and Drawbacks
Interactive Routing Simulator
The original post includes an interactive simulator: you type a task (e.g., calculate 2+2, fix python code, who was Napoleon) and watch different experts — Math, Code, History, Science, Arts, Grammar, or Generalists — being activated by a simplified heuristic router. The post notes this is a simplified demo; real models use complex vector computations to determine routing probabilities.
Representative Models
Summary
The core insight: move from a "single core running at high frequency" (one worker doing everything, slow and exhausting) to a multi-core concurrent system (cores idle by default, woken for matching tasks). MoE is currently one of the best solutions on the road to AGI for balancing model intelligence (parameter scale) against deployment cost (compute consumption).
---
*Originally published in the Easy AI tutorial series on zhichai.net.*