Geometric Algebra Rebuilds Deep Learning: A Dual Revolution in Low-Rank Approximation and Attention Mechanisms

Forum topic · 小凯 · 2026-05-18

Summary

A zhichai.net forum research review examines two papers that use Clifford (geometric) algebra to rethink core deep learning primitives. First, Pence et al. (NeurIPS 2025, arXiv:2507.11688) show that any linear function can be expressed as a sum of rotor sandwich products, parameterized by simple bivectors, yielding O(log²d) parameters versus O(rd) for SVD low-rank approximations. Replacing Q/K/V projections in LLaMA-3.2 1B with rotor layers (~896 parameters) achieves up to a 4700× parameter reduction while matching or beating rank-4 SVD and block-Hadamard baselines on Wikitext2, C4, and ARC benchmarks. Second, Hirst & Huy's Versor architecture (arXiv:2602.10195v2) introduces Geometric Product Attention, which decomposes similarity into scalar proximity plus bivector torque components, and a Recursive Rotor Accumulator that models sequence history in O(L) time via spins on the Spin manifold. On chaotic N-body dynamics, Versor achieves 200× fewer parameters than Transformers and reduces energy drift 150× (2.4% for a Hamiltonian variant), and on topological reasoning (Broken Snake) it reaches 0.993 MCC versus 0.070 for ViT. Custom Triton kernels deliver 78× speedup over naive PyTorch with 1.05 ms inference latency, beating a standard Transformer. Limitations include non-drop-in integration, fixed Cl(4,1) dimensionality, and weaker language modeling perplexity.

Key points

This forum research review surveys two papers that use Clifford (geometric) algebra to replace two foundational deep learning operations: low-rank matrix approximation and attention.

Part 1: Rotor-based linear layers as an alternative to SVD low-rank approximation

**Core result (Lemma 1, Pence et al., *Composing Linear Layers from Irreducibles*, NeurIPS 2025, arXiv:2507.11688): Any linear function can be written as a finite sum of sandwich products

F(x) = \sum_{t=1}^{w} a_t x b_t

where the factors are multivectors in a Clifford algebra. A particularly expressive form is the rotor sandwich product

\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n), \quad r = \exp(b) = \cos(\|b\|) + \frac{\sin(\|b\|)}{\|b\|} b

with b a simple bivector b = u ∧ v encoding a directed plane.

Differentiable decomposition: Two algorithms make the parameterization end-to-end trainable — a differentiable invariant decomposition of arbitrary bivectors into ⌊n/2⌋ orthogonal commuting simple bivectors, and a GA power-iteration using contractions b ⌟ v with a closed-form projection Proj_simple(b) = σ(u ∧ v).

Parameter complexity (Theorem 1): A map built from c₁ × c₂ rotor modules acting in Cl(n), with 2ⁿ ≤ min(d_in, d_out), needs at most

2c_1c_2\binom{n}{2} = O(\log^2 d)

parameters — e.g., ≤ 896 parameters for a LLaMA-3.2 1B query projection versus 4,194,304 dense (a 4700× reduction).

LLM experiments: Freezing a pretrained LLM and replacing layers sequentially (trained on extracted (x, y) pairs with MSE loss) on LLaMA-3.2 1B:

| Metric | Layers | Original | LR1 | LR4 | BH1 | Rotor | |:---|:---|:---|:---|:---|:---|:---| | Wikitext2 Log-PPL ↓ | 1 | 2.575 | 2.688 | 2.658 | 2.636 | 2.629 | | | 3 | 2.575 | 4.956 | 2.880 | 2.779 | 2.818 | | Arc Challenge ↑ | 1 | 58.37 | 50.78 | 53.84 | 54.83 | 55.31 | | HellaSwag ↑ | 1 | 41.00 | 36.17 | 38.02 | 39.10 | 39.33 |

Rotor consistently beats rank-4 SVD and block-Hadamard baselines in single- and double-layer replacement (also confirmed on Qwen-2.5 1.5B).

Insight: SVD assumes important information lies in a low-dimensional subspace; the geometric-algebra view instead decomposes linear transformations into compositions of geometric rotations, preserving geometric semantics with far fewer parameters. The authors note it is "not a drop-in replacement yet."

Part 2: Versor — Geometric Product Attention and O(L) sequence modeling

Scalar poverty of standard attention: Q·Kᵀ compresses high-dimensional multivector relationships into a single scalar, losing orientation/plane/volume information.

Geometric Product Attention (GPA)** (Hirst & Huy, *Versor*, arXiv:2602.10195v2) decomposes the geometric product into graded parts:

\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)

where the scalar grade measures proximity and the bivector grade measures "torque" (orientation), with learnable mixing coefficient γ.

Recursive Rotor Accumulator (RRA): sequence history as a single spin state on the Spin manifold, updated in O(L):

\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t), \quad \Psi\widetilde{\Psi} = 1

with O(1) memory instead of O(L²) attention matrices.

Chaotic 5-body dynamics: Versor (0.007M params) reaches MSE 5.210 vs Transformer's 6.609 (1.32M params); the Hamiltonian variant (Ham-Versor) achieves MSE 4.827 with only 2.4% energy drift vs 381.1% for Transformer — a 200× parameter reduction and 150× lower energy drift.
Topological reasoning (Broken Snake): MCC 0.993 vs 0.070 for ViT/Transformer; robust OOD performance (−63.9% error change at 10× mass vs +1933.7%); hidden-velocity inference MSE 0.003 vs 0.325 for GATr.
Hardware: A Triton/MLX bit-masked kernel exploiting XOR isomorphism gives 78× speedup over naive PyTorch; the Cl(4,1) ≅ Mat(4, ℂ) matrix isomorphism cuts latency 65%; a multithreaded C++ RRA core lowers end-to-end latency 7.5× — final inference latency 1.05 ms vs 1.10 ms for a Transformer.
Limitations: CGA is fixed at Cl(4,1) (32 dims); numerical drift requires manifold normalization; language modeling lags (WikiText perplexity 3.22 vs 2.5–2.7 for Transformers).

Unifying view

Both works exploit the graded structure of Clifford algebras Cl(p,q) = ⟨·⟩₀ ⊕ ⟨·⟩₁ ⊕ ⟨·⟩₂ ⊕ … ⊕ ⟨·⟩ₙ: rotor layers use bivectors (grade 2) to parameterize linear maps, while GPA uses the full geometric product to enrich attention scores. The common foundation is rotor = exp(bivector), i.e., Spin-group geometric transformations.

Directions for future work

1. End-to-end rotor-layer training inside LLM pipelines (not frozen replacement) 2. Higher-dimensional algebras (Cl(5,1), Cl(3,3)) for different tasks 3. Hybrid GPA + standard attention architectures 4. Large-scale hardware deployment beyond the existing 78× kernel speedup

References

1. Pence, T., Yamada, D., & Singh, V. (2025). *Composing Linear Layers from Irreducibles*. NeurIPS 2025. arXiv:2507.11688. 2. Hirst, E. & Huy, T. M. (2026). *Versor: A Geometric Sequence Architecture Enhanced Scale Generalization and Interpretability via Conformal Algebra*. arXiv:2602.10195v2. 3. Brehmer, J. et al. (2023). *Geometric Algebra Transformer*. NeurIPS 2023. https://github.com/Qualcomm-AI-research/geometric-algebra-transformer 4. Ginzberg, B. & Mavroyiakoumou, I. (2016). *SVD over Clifford Algebras*. Linear Algebra and its Applications. 5. Sadrzadeh, M. et al. (2026). *FGA for NLP*. arXiv:2604.25902. 6. Ji, S. (2026). *CliffordNet*. arXiv:2601.06793.