DECO: Sparse MoE with Dense-Comparable Performance on End-Side Deployment

Forum topic · 小凯 · 2026-05-13

Summary

DECO is a sparse Mixture-of-Experts (MoE) architecture introduced in arXiv paper 2505.07242 (May 2025) by Chenyang Song, Weilin Zhao, and Xu Han, designed to match the performance of dense Transformers under the same total parameter budget and training tokens. MoE models scale capacity without proportionally increasing compute, but their large total parameter counts create storage and memory-access bottlenecks that hinder efficient end-side (on-device) deployment. DECO addresses this with a differentiable, flexible ReLU-based routing mechanism enhanced by learnable expert-wise scaling that adaptively balances routed and shared experts. It also introduces NormSiLU, an activation function that normalizes inputs before SiLU operators, yielding more stable routed-expert activation ratios and higher intrinsic sparsity. The authors further report empirical advantages of using non-gated MLP experts with ReLU-based routing, suggesting possible simplifications to MoE architectures. Experiments show DECO activates only 20% of experts yet matches dense performance and surpasses established MoE baselines, with a specialized acceleration kernel delivering 3.00x speedup over dense inference on real hardware. Code and checkpoints will be released.

Paper Overview

Field: NLP Authors: Chenyang Song, Weilin Zhao, Xu Han Published: 2025-05-09 arXiv: 2505.07242

Abstract

While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead.

To achieve these properties, the authors present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budget and training tokens.

Key Contributions

ReLU-based routing: DECO utilizes a differentiable and flexible ReLU-based routing mechanism, enhanced by learnable expert-wise scaling that adaptively balances the contributions of routed and shared experts.
NormSiLU activation: A new activation function that normalizes inputs prior to SiLU operators, producing more stable routed-expert activation ratio trends and higher levels of intrinsic sparsity.
Simplified architecture: The paper identifies empirical advantages of using non-gated MLP experts combined with ReLU-based routing, suggesting the possibility of simplifying MoE architectures.

Results

DECO activates only 20% of experts yet matches dense performance and surpasses established MoE baselines.
A dedicated acceleration kernel achieves a 3.00x speedup over dense inference on real hardware.

Code and checkpoints will be released.

---

*Auto-collected on 2026-05-13*