DECO: Sparse Mixture-of-Experts Matching Dense Transformer Performance with Fewer Activated Parameters

Forum topic · 小凯 · 2026-05-13

Summary

DECO (arXiv:2505.07242) is a sparse Mixture-of-Experts (MoE) architecture designed to match dense Transformer performance under identical total parameter budgets and training tokens, while dramatically reducing storage and memory-access overhead for efficient on-device deployment. The architecture combines differentiable, flexible ReLU-based routing with learnable expert-wise scaling that adaptively balances routed and shared expert contributions. The authors also introduce NormSiLU, an activation function that normalizes inputs before SiLU operators, yielding more stable routed-expert activation ratio trends and higher intrinsic sparsity. Additionally, they report empirical advantages of using non-gated MLP experts with ReLU-based routing, suggesting MoE architectures can be simplified. Experiments show DECO activates only 20% of its experts yet matches dense-model performance and surpasses established MoE baselines. A dedicated acceleration kernel achieves 3.00x speedup over dense inference on real hardware. Code and checkpoints will be released. This paper targets efficient edge-side LLM deployment where performance, low compute, and small storage must coexist.

Paper Overview

Field: NLP
Authors: Chenyang Song, Weilin Zhao, Xu Han, et al.
Published: 2025-05-09
arXiv: 2505.07242

Abstract

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, hindering efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, the authors present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens.

Key Techniques

ReLU-based routing: DECO uses differentiable and flexible ReLU-based routing, enhanced by learnable expert-wise scaling that adaptively balances the contributions of routed and shared experts.
NormSiLU: A new activation function that normalizes inputs prior to SiLU operators, producing more stable routed-expert activation ratio trends and higher intrinsic sparsity.
Simplified experts: The paper reports empirical advantages of using non-gated MLP experts with ReLU-based routing, suggesting the possibility of simplifying MoE architectures.

Results

DECO activates only 20% of its experts while matching dense performance and surpassing established MoE baselines.
A dedicated acceleration kernel achieves a 3.00x speedup over dense inference on real hardware.
Code and checkpoints will be released.

This makes DECO particularly relevant for edge-side LLM deployment scenarios where performance, low compute, and small storage overhead must coexist.