Background: Why MoE matters for on-device inference
Mixture-of-Experts (MoE) models such as Mixtral 8x7B keep total parameters large (about 47B) while activating a subset per forward pass (about 12.9B effective parameters). Compute scales with active parameters, not total parameters, so an MoE can deliver dense-quality capacity at lower inference cost. The catch is training: converting a dense LLM such as Llama-2 7B into MoE has traditionally required continual pre-training, auxiliary-load-balancing losses, and careful tuning of routing hyperparameters — none of which are practical on edge hardware.
What CMoE does
CMoE (Converting Mixture-of-Experts from Dense) is a training-free pipeline that:
1. Profiles FFN neurons with a small calibration dataset and records activation frequencies. 2. Assigns high-frequency neurons to a shared expert (always active) and low-frequency neurons to routed experts. 3. Solves a balanced expert-assignment problem with the Jonker-Volgenant shortest-path-augmentation algorithm so each expert has coherent activation patterns and even load. 4. Builds a differentiable router directly from activation statistics — router weights correlate input tokens with each expert's representative neurons — so the router is meaningful before any gradient step.
The full conversion takes about five minutes on a single GPU.
Experimental results
Language modeling perplexity (zero training)
| Method | WikiText-2 PPL | C4 PPL | |---|---|---| | Llama-2 7B dense | ~7.5 | ~12.5 | | LLaMA-MoE (no training) | > 20,000 / NaN | collapsed | | LLaMA-MoE-v2 (no training) | > 20,000 / NaN | collapsed | | CMoE 75% active | ~7.5 (lossless) | ~12.5 (lossless) | | CMoE 50% active | ~8.5 | ~13.5 | | CMoE 25% active | ~60 | ~45 |
Random neuron splitting causes the baseline MoE conversions to collapse; activation-aware grouping avoids that failure mode.
Downstream tasks after 1-hour LoRA on 2,000 samples (25% active CMoE)
| Task | Dense Llama-2 7B | CMoE 25% + LoRA | Recovery | |---|---|---|---| | BoolQ | 76.2 | ~70 | 91.8% | | PIQA | 79.5 | ~75 | 94.3% | | SciQ | 93.5 | ~86 | 92.0% | | Winogrande | 72.1 | ~68 | 94.3% | | ARC-C | 54.2 | ~48 | 88.6% | | HellaSwag | 78.3 | ~72 | 92.0% | | Average | 75.6 | ~69.8 | ~92.2% |
The paper reports >76% recovery of dense-model downstream accuracy at 25% activation with one hour of LoRA fine-tuning.
Latency
| Config | MLP speedup | End-to-end speedup | |---|---|---| | S1A1E8 (25% active) | 2.0–2.2x | 1.4–1.6x | | S1A3E8 (50% active) | 1.4–1.5x | 1.2–1.3x | | S1A3E4 (75% active) | 1.1–1.2x | 1.05x |
End-to-end numbers are smaller than MLP-only numbers because attention layers and routing overhead are fixed costs.
Why it works
- Implicit functional partitioning already exists in dense FFNs; activation analysis surfaces it instead of inventing it.
- Routing is recall, not learning — the router's weights come from observed activations, so they are accurate from the first token.
- Shared experts act as a safety net, preserving core language capability when routed experts are mis-selected, which is why CMoE does not collapse without training.
- For 7B-class LLMs, FFNs are roughly 70% of parameters; activating 25% of FFN neurons saves about 52.5% of FFN memory bandwidth per token.
- Conversion and one-hour LoRA recovery fit on a single GPU, so edge-server operators can tune activation rate (25% / 50% / 75%) per hardware budget.
- Model vendors can keep one dense checkpoint and ship either dense or MoE deployments without re-training.
- Training-free Dense-to-MoE conversion in ~5 minutes using only neuron activation statistics.
- 75% activation yields lossless WikiText-2 / C4 perplexity; 25% activation delivers up to 1.5x end-to-end speedup.
- One-hour LoRA on 2,000 samples recovers ~92% of dense downstream accuracy across six benchmarks.
- Jonker-Volgenant balanced assignment plus analytic differentiable router differentiate CMoE from random-split or importance-split MoE conversions.
- Validated on Llama-2 7B and Llama-3 8B; relevant to mobile, edge-server, and dual dense/MoE deployment strategies.
- Paper: CMoE: Converting Mixture-of-Experts from Dense to Accelerate LLM Inference
- Authors: Zehua Pei, Lancheng Zou, Hui-Ling Zhen, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu
- Institutions: The Chinese University of Hong Kong; Huawei Noah's Ark Lab
- arXiv: 2502.04416
- Code: https://github.com/JarvisPei/CMoE
Comparison with related work
| Method | Training needed | Conversion time | PPL recovery | Router construction | |---|---|---|---|---| | CMoE | None | ~5 min | 75% lossless | Analytic from activations | | LLaMA-MoE | Continual pre-training | Days–weeks | After training | Random init + train | | LLaMA-MoE-v2 | Importance-based split + training | Hours | After training | Hidden-feature based | | Sparse Up-cycling | Training required | Hours–days | After training | Random init + train | | MoEfication | Training required | Hours | After training | Cosine-similarity based |
CMoE is the only fully training-free Dense-to-MoE pipeline in this comparison.
On-device implications
Limitations
1. At 25% activation without fine-tuning, perplexity degrades sharply (PPL ~60 on WikiText-2); LoRA is required for usable quality. 2. Expert grouping depends on calibration-data distribution. 3. Only FFN layers are converted; attention compute is unchanged. 4. Real end-to-end speedup depends on hardware sparse-compute support. 5. Evaluation is on English text only; multilingual and multimodal behavior is open.