[论文] Disentangling Representation Evolution in Transformers through Directi...

论文概要

研究领域: NLP 作者: Shwai He, Haichao Zhang, Shen Yan 发布时间: 2026-09-14 arXiv: 2609.15975

中文摘要

Transformer 的表征通过学习到的加性变换演化——这些变换要么保持当前方向，要么重定向。我们将这种演化视为函数几何，将学习到的更新分解为平行和垂直分量。在预训练模型中，我们发现除残差恒等路径外还存在显著的平行分量。然后我们在两个空间中应用该分解：注意力/MLP 更新相对于隐藏状态，以及注意力值聚合相对于当前令牌的值。定向编辑揭示了一个强烈的空间依赖性不对称：排除自身的值空间平行操作明显比残差空间和垂直对应操作更鲁棒，它在保留直接自身消息的同时仅缩放非自身聚合。同一分解还给出压缩引起的更新误差的分量级描述：垂直误差比平行误差更清晰地区分了不同的压缩方法。大量实验进一步表明，从零开始预训练期间的全聚合平行抑制降低了验证损失轨迹并改善了下游平均表现，其中值空间变体效果最强。这些结果将表征几何与编辑鲁棒性、压缩诊断和训练时干预联系起来。代码见项目仓库。

原文摘要

Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study this evolution as a functional geometry, decomposing learned updates into parallel and perpendicular components. Across pretrained models, we find substantial parallel components beyond the residual identity path. We then apply the decomposition in two spaces: to attention and MLP updates relative to the hidden state, and to attention value aggregation relative to the current token's value. Targeted edits reveal a strongly space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate....

#论文 #arXiv #NLP #小凯