[论文] DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on E...

## 论文概要 **研究领域**: NLP **作者**: Chenyang Song, Weilin Zhao, Xu Han **发布时间**: 2025-05-09 **arXiv**: [2505.07242](https://arxiv.org/abs/2505.07242) ## 中文摘要 虽然专家混合（MoE）在不成比例增加计算的情况下扩展了模型容量，但其巨大的总参数量造成了显著的存储和内存访问瓶颈，阻碍了需要高性能、低计算成本和小存储开销的端侧高效部署。为实现这些特性，我们提出了DECO，一种稀疏MoE架构，旨在在相同的总参数预算和训练token下匹配密集Transformer的性能。DECO利用可微且灵活的基于ReLU的路由，通过可学习的专家级缩放增强，自适应地平衡路由专家和共享专家的贡献。此外，我们引入了NormSiLU，一种在SiLU算子之前归一化输入的激活函数，产生更稳定的路由专家激活比趋势和更高的内在稀疏性水平。我们还发现了使用非门控MLP专家配合基于ReLU的路由的经验优势，表明MoE架构简化的可能性。实验表明，DECO仅激活20%的专家，即可匹配密集性能并超越已建立的MoE基线。我们的专用加速内核在真实硬件上相比密集推理实现了3.00倍加速。代码和检查点将发布。 ## 原文摘要 While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total参数预算和训练token。DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing ... --- *自动采集于 2026-05-13* #论文 #arXiv #NLP #小凯