The Token Is a Group Element: Lie-Algebra Attention over Matrix Lie Groups

Forum topic · 小凯 · 2026-06-22

Summary

This arXiv paper (2506.17582) by Przemyslaw Musialski proposes a novel attention mechanism in which each token is a raw element of a matrix Lie group—a bare transformation with no feature payload or group representation attached. Attention scores are computed as closed-form algebraic norms of relative poses: s_ij = -||log(g_i^{-1} g_j)||_lambda^2 / tau, a canonical proximity kernel under a block-weighted Frobenius inner product. This eliminates the need for irreducible representations, spherical harmonics, Clebsch-Gordan machinery, or learned kernels. Pairwise invariants arise intrinsically, equivariance under the diagonal group action is automatic, and the cocycle condition holds naturally. The construction applies to any matrix Lie group whose log chart contains relative poses, including the non-compact, non-abelian affine group with scale and shear—out of reach for vector-token attention based on irreducible representations or surjective exponential maps. Experiments on sequence completion over SE(2), SO(3), and Aff(2) show that closed-form scores match or beat learned MLP kernels (better on SE(2)) with 50-80x fewer score parameters, while vector-token baselines break equivariance by 5-12 orders of magnitude.

Paper: arXiv:2506.17582 — Przemyslaw Musialski, cs.LG / cs.CV / cs.GR, 2026-06-21

Core Idea

The paper proposes Lie-algebra attention, where the attention token is a group element: each token is an element $$g_i$$ of a matrix Lie group $$G$$ — a bare transformation with no feature payload and no external representation $\rho(g)$ carrying it. To the author's knowledge, this is the first attention construction using bare matrix Lie group elements as tokens.

Key Properties

Canonical pairwise geometry: relative poses $g_i^{-1} g_j$ , so pairwise invariants $w_{ij} = \log(g_i^{-1} g_j)$ are intrinsic rather than designed.
Automatic equivariance: equivariance under the diagonal $$G$$ action is immediate, and the cocycle condition holds naturally — no representation-theoretic machinery required.
Closed-form scores:

s_{ij} = -\|\log(g_i^{-1} g_j)\|_\lambda^2 / \tau

a canonical proximity kernel under a block-weighted Frobenius inner product. No irreducible representations, spherical harmonics, Clebsch-Gordan products, or learned kernels.

Broad applicability: works for any matrix Lie group whose log chart contains relative poses — including the non-compact, non-abelian affine group with scale and shear, which is excluded by all methods based on irreducible representations or surjective exponential maps, and is out of reach for any vector-token attention approach.

Experimental Results

Three sequence-completion experiments over SE(2), SO(3), and Aff(2) support the thesis:

Closed-form scores match learned MLP kernels on the same invariants, and outperform them on SE(2).
The approach uses 50–80x fewer score parameters.
Vector-token baselines break equivariance by 5–12 orders of magnitude.

---

*Auto-collected 2026-06-21.*