Paper Overview
Field: ML Author: Przemyslaw Musialski Published: 2026-06-20 arXiv: 2506.16541
Abstract
We put the attention token on the group: the token is an element g_i of a matrix Lie group G—a bare transformation, with no feature payload and no carrying external action rho(g). To our knowledge, this is the first attention construction whose tokens are bare matrix Lie group elements: the scores are closed-form algebraic norms of relative poses, rather than learned kernels, and it reaches the full affine frame group—something every construction based on irreducible representations or a surjective exponential must exclude. We call it Lie-algebra attention.
Once the token is a group element, the rest requires none of the usual representation-theoretic machinery. The pairwise relative geometry is canonical, g_i^{-1} g_j, so the pairing invariant w_ij = log(g_i^{-1} g_j) is intrinsic rather than designed; equivariance under the diagonal G-action is tautological, and the cocycle condition holds automatically. The attention score is the negative squared algebraic norm, s_ij = -||log(g_i^{-1} g_j)||_lambda^2 / tau: a canonical proximity kernel under a block-weighted Frobenius inner product, with no irreducible representations, spherical harmonics, Clebsch-Gordan products, or learned kernels.
The construction applies to any matrix Lie group on a chosen log chart that includes relative poses, including the non-compact, non-abelian affine group with scaling and shear—which no vector-token attention method can reach, whether from the irreducible-representation tradition or the surjective-exponential approach. Three sequence-completion experiments, on SE(2), SO(3), and Aff(2), confirm this: the closed-form scores match learned MLP kernels on the same invariants and surpass them on SE(2), using 50 to 80 times fewer score parameters, while vector-token baselines break invariance by five to twelve orders of magnitude.
--- *Auto-collected on 2026-06-21*