[论文] Routers Learn the Geometry of Their Experts: Geometric Coupling in Spa...

## 论文概要 **研究领域**: NLP **作者**: Sagi Ahrac, Noya Hochwald, Mor Geva **发布时间**: 2026-05-12 **arXiv**: [2605.12476](https://arxiv.org/abs/2605.12476) ## 中文摘要 稀疏专家混合（SMoE）模型高效扩展语言模型，但训练它们仍然具有挑战性，路由可能崩溃到少数专家，辅助负载均衡损失可能降低专业化。受这些障碍驱动，我们研究 SMoE 中路由决策如何机制性形成。首先，我们揭示路由器和对应专家之间的几何耦合。对于给定 token，所选专家的路由器权重和处理它的专家权重沿相同输入方向接收梯度，仅标量系数不同。因此匹配的路由器-专家方向累积相同的路由 token 历史。此理论耦合也出现在路由动态的经验中。在从头训练的 1B SMoE 中，更高路由器分数预测更强专家神经元激活，表明路由决策在选定专家内部被镜像。接下来，我们分析辅助负载均衡对路由器-专家几何耦合的影响，表明此类损失通过在路由器权重上扩散输入定向梯度来破坏此结构，使不同路由器方向彼此相似近三倍。最后，我们通过无参数的在线 K-Means 路由器展示几何耦合对有效路由的中心性，其中每个专家维护路由到它的隐藏状态的运行平均值，token 基于余弦相似度分配。与辅助损失和无损失平衡相比，此路由器以仅适度的困惑度增加实现最低负载不平衡，表明几何耦合捕获了路由器学习的大部分内容。 ## 原文摘要 Sparse Mixture-of-Experts (SMoE) models enable scaling language models efficiently, but training them remains challenging, as routing can collapse onto few experts and auxiliary load-balancing losses can reduce specialization. Motivated by these hurdles, we study how routing decisions in SMoEs are formed mechanistically. First, we reveal a geometric coupling between routers and their corresponding experts. For a given token, the router weights for the selected expert and the expert weights processing it receive gradients along the same input direction, differing only in scalar coefficients. Thus, matched router--expert directions accumulate the same routed token history. This theoretical coupling also appears empirically in routing dynamics. In a 1B SMoE trained from scratch, higher router... --- *自动采集于 2026-05-14* #论文 #arXiv #NLP #小凯