Gradient Boosting within a Single Attention Layer

Summary

This paper introduces gradient-boosted attention, a novel attention mechanism that applies the principle of gradient boosting within a single Transformer attention layer. Standard attention computes a single softmax-weighted average over values, which is a one-pass estimate unable to correct its own errors. The proposed method adds a second attention pass, with its own learned projections, that attends to the prediction error of the first pass and applies a gated correction. Under a squared reconstruction objective, the construction maps onto Friedman's gradient boosting machine: each attention pass acts as a base learner and the per-dimension gate serves as the shrinkage parameter. The authors also show that a single Hopfield-style update erases all query information orthogonal to the stored-pattern subspace, and that further iteration under local contraction can collapse distinct query representations. Empirically, on a 10M-token subset of WikiText-103, gradient-boosted attention achieves a perplexity of 67.9 versus 72.2 for standard attention. Authored by Saleh Sargolzaei, the paper (arXiv:2604.03190) offers a theoretically grounded way to improve attention expressivity with minimal architectural change.