The Token Is a Group Element: Lie-Algebra Attention over Matrix Lie Groups

Forum topic · 小凯 · 2026-06-22

Summary

This post summarizes arXiv paper 2506.17582 by Przemyslaw Musialski, which proposes a novel attention mechanism in which tokens are raw elements of a matrix Lie group rather than feature vectors. Attention scores are computed as closed-form algebraic norms of relative poses, s_ij = -||log(g_i^{-1} g_j)||^2 / tau, using a block-weighted Frobenius inner product instead of learned kernels, irreducible representations, spherical harmonics, or Clebsch-Gordan machinery. Pairwise invariants arise canonically from the relative geometry g_i^{-1} g_j, equivariance under the diagonal group action is automatic, and the cocycle condition is naturally satisfied. The method applies to any matrix Lie group where the chosen log chart covers relative poses, including the non-compact, non-abelian affine group Aff(n) containing scale and shear, which vector-token methods cannot reach. Sequence completion experiments on SE(2), SO(3), and Aff(2) show that the closed-form scores match learned MLP kernels on the same invariants, outperform them on SE(2) with 50-80x fewer score parameters, while vector-token baselines break equivariance by five to twelve orders of magnitude.

Paper Overview

Field: cs.LG, cs.CV, cs.GR Author: Przemyslaw Musialski Published: 2026-06-21 arXiv: 2506.17582

Summary

This paper introduces Lie-algebra attention, where each attention token is an element $$g_i$$ of a matrix Lie group $$G$$ — a bare transformation carrying no feature payload and no external representation action $\rho(g)$ . To the authors' knowledge, this is the first attention construction that uses bare matrix Lie group elements as tokens.

Key points

Tokens as group elements: Each token is a raw transformation in a matrix Lie group $$G$$ , with no feature load and no external action $\rho(g)$ .
Canonical pairwise geometry: Relative poses are given by $g_i^{-1} g_j$ , so the pairwise invariant $w_{ij} = \log(g_i^{-1} g_j)$ is intrinsic rather than designed. Equivariance under the diagonal $$G$$ action is automatic, and the cocycle condition is naturally satisfied.
Closed-form attention scores: Scores take the form

s_{ij} = -\|\log(g_i^{-1} g_j)\|_\lambda^2 / \tau

a canonical proximity kernel under a block-weighted Frobenius inner product — no irreducible representations, spherical harmonics, Clebsch–Gordan products, or learned kernels required.

Broad applicability: The construction works for any matrix Lie group, provided the chosen log chart covers relative poses — including the non-compact, non-abelian affine group, which contains scale and shear and is unreachable by any vector-token attention method, whether based on irreducible representations or surjective exponential maps.
Experimental validation: Three sequence completion experiments on SE(2), SO(3), and Aff(2) show that the closed-form scores match learned MLP kernels on the same invariants, outperform them on SE(2) with 50–80× fewer score parameters, while vector-token baselines break equivariance by five to twelve orders of magnitude.

---

*Auto-collected on 2026-06-21*

The Token Is a Group Element: Lie-Algebra Attention over Matrix Lie Groups

Summary

Paper Overview

Summary

Key points

Tags