[论文] Quantifying Concentration Phenomena of Mean-Field Transformers in the ...

## 论文概要 **研究领域**: ML **作者**: Albert Alcalde, Leon Bungert, Konstantin Riedl **发布时间**: 2025-05-09 **arXiv**: [2505.07241](https://arxiv.org/abs/2505.07241) ## 中文摘要 以自注意力模块为核心组件的Transformer已成为现代大型语言和基础模型的核心架构。本文研究深度仅编码器Transformer在推理时token的演化，这在大量token极限下由平均场连续性方程描述。借鉴相互作用多粒子系统（粒子对应token）收敛分析的思想，我们证明token分布迅速集中到由key、query和value矩阵诱导的投影映射下初始分布的前推，并在中等时间内保持亚稳态。具体地，我们证明两个分布的Wasserstein距离按温度参数beta^(-1)->0和推理时间t>=0缩放为sqrt(log(beta+1)/beta)*exp(Ct)+exp(-ct)。为证明这一点，我们对零温度方程建立了Lyapunov型估计，确定了t->无穷时的极限，并在Wasserstein空间中使用了稳定性估计结合定量Laplace原理来耦合两个方程。我们的结果表明，对于log(beta)阶的时间尺度，token分布在确定的极限分布处集中。数值实验证实了这一点，并进一步补充了我们的理论，表明对于有限beta和大t，动态进入不同的终端阶段，由value矩阵的谱主导。 ## 原文摘要 Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multi-particle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like sqrt(log(beta+1)/beta) * exp(Ct)... --- *自动采集于 2026-05-13* #论文 #arXiv #ML #小凯