Training Communication-Efficient Mixture-of-Experts Language Models (CE-MoE) — arXiv 2608.28511

Forum topic · 小凯 · 2026-09-01

Summary

This paper introduces CE-MoE, a communication-efficient approach to training Mixture-of-Experts (MoE) language models. When using expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. CE-MoE adopts a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth: instead of interleaving MoE layers after every token-mixing layer (e.g., attention or Mamba-2), expert capacity is concentrated in a select few routed MoE layers, while model depth is preserved by adding extra token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE consistently reduces training cost while matching validation loss and downstream benchmarks of fully interleaved MoE baselines. At 31.5B parameters, CE-MoE uses 33.3% fewer GPU-hours while improving average downstream scores and inference throughput. Paper by Simeng Sun and Roger Waleffe, arXiv:2608.28511.

Paper Overview

Field: Machine Learning
Authors: Simeng Sun, Roger Waleffe
Published: 2026-08-28
arXiv: 2608.28511

Abstract

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, the authors study communication-efficient MoE models (CE-MoE), which adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth.

Key Idea

Conventional models interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2). CE-MoE instead:

Concentrates expert capacity in a select few routed MoE layers
Maintains model depth by adding additional token-mixing and dense-FFN layers
Reduces the frequency of costly all-to-all communication

Results

Evaluated on a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters
CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks of full-MoE baselines
At 31.5B scale: 33.3% fewer GPU-hours, improved average downstream scores, and higher inference throughput