English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Paper: Training Communication-Efficient Mixture-of-Experts Language Models (CE-MoE)

Forum topic · 小凯 · 2026-09-01

Summary

This paper (arXiv:2608.28511) by Simeng Sun and Roger Waleffe introduces CE-MoE, a communication-efficient approach to training Mixture-of-Experts language models. When training MoE models with expert parallelism, all-to-all token dispatch and combine collectives consume a substantial fraction of end-to-end training time. CE-MoE adopts a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth: instead of interleaving MoE layers after every token-mixing layer (such as attention or Mamba-2), expert capacity is concentrated in a select few routed MoE layers, while model depth is maintained by adding extra token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, with matched total and activated parameter counts, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks of full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3% fewer GPU hours while improving average downstream scores and inference throughput.

Paper Overview

  • Research area: Machine Learning
  • Authors: Simeng Sun, Roger Waleffe
  • Published: 2026-08-28
  • arXiv: 2608.28511

    • Summary

    When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, the authors study communication-efficient MoE models (CE-MoE), adopting a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth.

    Unlike conventional models that interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers.

    Key Results

  • Evaluated across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters.
  • CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmark performance of full-MoE baselines.
  • At the 31.5B scale, CE-MoE uses 33.3% fewer GPU hours while improving average downstream scores and inference throughput.

Original Abstract (excerpt)

> When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth...

---

*Auto-collected on 2026-09-01. Source: arXiv:2608.28511*

Tags

#mixture-of-experts#communication-efficient-training#language-models#expert-parallelism#arxiv#machine-learning#training-efficiency

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178634350