Summary
This arXiv paper (2608.28511) by Simeng Sun and Roger Waleffe introduces CE-MoE, a communication-efficient Mixture-of-Experts architecture for language model training. When training MoE models with expert parallelism, all-to-all token dispatch and combine collectives can consume a large fraction of end-to-end training time. CE-MoE addresses this with a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth: instead of interleaving MoE layers after every token-mixing layer (attention, Mamba-2), expert capacity is concentrated in a select few routed MoE layers, with depth preserved by adding extra token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, with matched total and activated parameters, CE-MoE consistently reduces training cost while matching validation loss and downstream benchmarks of full-MoE baselines. At 31.5B scale, CE-MoE uses 33.3% fewer GPU hours while improving average downstream scores and inference throughput.
Paper Overview
- Field: Machine Learning
- Authors: Simeng Sun, Roger Waleffe
- Published: 2026-08-28
- arXiv: 2608.28511
Summary
When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. This work studies communication-efficient MoE models (CE-MoE), which adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth.
Compared to conventional models that interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers.
Key Results
- Evaluated across a scaling ladder from 2B to 31.5B total parameters.
- Under matched total and activated parameters, CE-MoE models consistently reduce training cost.
- Match validation loss and downstream benchmarks of full-MoE baselines.
- At the 31.5B scale, CE-MoE uses 33.3% fewer GPU hours while improving average downstream scores and inference throughput.
---
*Auto-collected on 2026-09-01.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178634350