Paper Overview
Research area: Machine Learning Author: Przemyslaw Musialski Published: 2026-06-20 arXiv: 2506.16541
Abstract (translated)
We place attention tokens on a group: a token is an element g_i of a matrix Lie group G — a bare transformation with no feature payload and no external representation rho(g) attached. To our knowledge, this is the first attention construction in which the token is a bare matrix Lie group element: its scores are closed-form algebraic norms of relative pose rather than learned kernels, and it reaches the full affine framework group, which every approach based on irreducible representations or surjective exponentials must exclude. We call this Lie-algebra attention.
Once the token is a group element, everything else follows without any of the usual representation-theoretic machinery. The relative geometry of a pair is canonical, g_i^{-1} g_j, so the pairwise invariant w_ij = log(g_i^{-1} g_j) is intrinsic rather than designed; equivariance under the diagonal G-action is a tautology, and the co-ring condition holds automatically.
The attention score is the negative squared algebra norm, s_ij = -||log(g_i^{-1} g_j)||_lambda^2 / tau — a canonical proximity kernel under a block-weighted Frobenius inner product, with no irreducible representations, spherical harmonics, Clebsch-Gordan products, or learned kernels. The construction applies to any matrix Lie group, on chosen logarithm maps that include relative pose, including the non-compact, non-abelian affine group with scaling and shear — unreachable by any vector-token attention method, whether from the irreducible-representation tradition or the surjective-exponential family.
Three sequence-completion experiments, on SE(2), SO(3), and Aff(2), confirm this: the closed-form scores match learned MLP kernels on the same invariants and outperform them on SE(2), using 50 to 80 times fewer score parameters, while vector-token baselines break invariance by five to twelve orders of magnitude.
---
*Auto-collected on 2026-06-21*