How to Loop MoE: Flatten the Experts, Untie the Attention — Foil (arXiv 2609.35751)

Forum topic · 小凯 · 2026-09-30

Summary

This paper introduces Foil, a method for combining looped Transformers with sparse mixture-of-experts (MoE) models. Looped Transformers reuse one block of layers multiple times to push a fixed-size model further with extra computation, while sparse MoE activates only a few experts per token. Foil answers the question of how to loop a MoE with two design choices: (1) flattening the experts—halving the number of expert layers, doubling experts per layer, and doubling the passes so each routing decision selects from a larger pool—and (2) untying the attention, giving each pass its own attention parameters while experts and routers remain shared. With expert parameters and per-token expert compute held fixed, experiments show Foil clearly outperforms unflattened looped baselines: lower pretraining loss at 20B tokens, monotonically improving loss with more flattening at 100B tokens (best model 0.012 nats below baseline), and equal-or-better downstream accuracy. Untied attention also yields more balanced, confident routing. The paper offers design guidance: loop and widening benefits amplify each other, and routing confidence tracks healthy expert usage better than load balancing.

Paper Overview

Field: NLP Authors: Shouren Wang, Chuang Ma, Mohsen Hariri, Debargha Ganguly, Wang Yang, Xiaoqing Tong, Qianying Liu, Xiaotian Han, Vipin Chaudhary arXiv: 2609.35751

Abstract (Original)

Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a question: how to loop a MoE? We answer it with Foil. With the expert parameters and the expert compute per token held fixed, Foil (1) flattens the experts, halving the expert layers, doubling the experts per layer and doubling the passes, so that every routing decision chooses from a larger pool, and (2) unties the attention, giving each pass its own attention parameters while the experts and routers stay shared.

Key Findings

Flattening works: At 20B tokens, every Foil model achieves lower pretraining loss than unflattened looped baselines. At 100B tokens, loss improves monotonically with the degree of flattening; the flattest Foil reaches 0.012 nats below the baseline at equal parameters and compute, with downstream accuracy on par or better.
Untied attention helps routing: Giving each pass independent attention parameters (while experts and routers stay shared) produces more balanced and more confident routing at the same model shape.
Design guidance: The benefits of looping and widening expert layers amplify each other. Routing confidence tracks healthy expert usage better than load balancing, so sparse looped MoE models should use more experts per layer and more passes.

*Auto-collected on 2026-09-30 from zhichai.net.*