[论文] Gradient Boosting within a Single Attention Layer

论文概要

研究领域: ML 作者: Saleh Sargolzaei 发布时间: 2026-04-03 arXiv: 2604.03190

中文摘要

Transformer注意力计算对值的单一softmax加权平均——一个无法纠正自身错误的一次性估计。我们引入梯度增强注意力，它将梯度增强原理应用于单个注意力层内：第二个具有自己学习投影的注意力传递，关注第一个的预测误差并应用门控校正。在WikiText-103上的10M token子集上，梯度增强注意力达到67.9的困惑度，而标准注意力为72.2。

原文摘要

Transformer attention computes a single softmax-weighted average over values -- a one-pass estimate that cannot correct its own errors. We introduce \emph{gradient-boosted attention}, which applies the principle of gradient boosting \emph{within} a single attention layer: a second attention pass, with its own learned projections, attends to the prediction error of the first and applies a gated correction. Under a squared reconstruction objective, the construction maps onto Friedman's gradient boosting machine, with each attention pass as a base learner and the per-dimension gate as the shrinkage parameter. We show that a single Hopfield-style update erases all query information orthogonal to the stored-pattern subspace, and that further iteration under local contraction can collapse distin...

#论文 #arXiv #ML #小凯