Geometric Algebra Reshapes Deep Learning: Low-Rank Approximation and Attention Reinvented

Forum topic · 小凯 · 2026-05-18

Summary

This forum post surveys two recent papers that use Clifford (geometric) algebra to redesign core deep learning primitives. First, Pence et al. (NeurIPS 2025, arXiv:2507.11688) replace SVD-style low-rank factorization with a rotor-based parameterization: any linear map can be written as a sum of sandwich products F(x) = sum a_t x b_t, yielding O(log^2 d) parameters versus O(d^2) for dense layers. On LLaMA-3.2 1B, replacing Q/K/V projections layer-by-layer with rotor modules achieves up to ~4700x parameter reduction while matching or beating rank-4 SVD and Block-Hadamard baselines in perplexity and downstream accuracy. Second, Hirst & Huy (arXiv:2602.10195) propose Versor, a Geometric Sequence Architecture using Geometric Product Attention (scalar proximity plus bivector torque terms) and a Recursive Rotor Accumulator with O(L) sequence cost. Versor reportedly cuts parameters 200x on chaotic N-body dynamics, reduces energy drift 150x (2.4% with a Hamiltonian variant), and generalizes zero-shot on topology tasks (0.993 vs 0.070 MCC) where Transformers collapse, backed by a Triton/MLX kernel delivering up to 78x speedup. The post also discusses shared mathematical foundations, current limitations, and future directions.

Geometric Algebra Reshapes Deep Learning: A Dual Revolution in Low-Rank Approximation and Attention Mechanisms

> Survey goal: Answer two questions — (1) Can classical SVD low-rank factorization be replaced by geometric-algebra parameterizations? (2) Can the efficient rotations of rotors fundamentally redesign attention computation? > > Core papers: > - Pence et al., *Composing Linear Layers from Irreducibles*, NeurIPS 2025 (arXiv:2507.11688) > - Hirst & Huy, *Versor: A Geometric Sequence Architecture*, arXiv:2602.10195v2

Part 1: A New Take on Low-Rank Approximation

Problem statement

Classical SVD low-rank decomposition:

W ≈ U_r Σ_r V_r^T, with O(r(d_in + d_out)) parameters
Essentially finds the best r-dimensional subspace in a vector space
Limitation: r is a hyperparameter; truncating singular values loses geometric structure

The Clifford algebra answer: linear maps = compositions of bivectors

Core math (Lemma 1): Any linear function can be expressed as a finite sum of sandwich products of bivectors:

F(x) = \sum_{t=1}^{w} a_t x b_t

where a_t, b_t are multivectors in a Clifford algebra and w < ∞ is the width.

More specifically, the rotor sandwich product:

\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n)

with r = exp(b), where b is a simple bivector (b = u ∧ v, the outer product of two vectors encoding an oriented plane).

Parameterization:

A simple bivector has \binom{n}{2} scalar coefficients (all planes in n-dimensional space)
Rotors via exponential map: r = cos(||b||) + sin(||b||)/||b|| · b
Multiple rotors aggregate via pooling: ψ(x) = σ({ψ_{r_{ij},s_{ij}}(x^{I_i})})

Differentiable decomposition: from bivector to rotor

The key engineering challenge. Two algorithms are proposed:

Algorithm 1 (differentiable invariant decomposition): given b ∈ Cl_2(n), extract k = ⌊n/2⌋ mutually commuting, orthogonal simple bivectors, iteratively avoiding eigendecomposition instability.
Algorithm 2 (GA power iteration): replaces matrix-vector products with right contractions b ⌞ v; closed-form projection Proj_simple(b) = σ(u ∧ v); convergence check uses sums instead of differences (sign symmetry).

This makes the rotor parameterization end-to-end differentiable and autograd-compatible.

Parameter efficiency: an O(log² d) guarantee

Theorem 1: For ψ: R^{d_in} → R^{d_out} composed of c_1 × c_2 rotor modules acting in Cl(n) with 2^n ≤ min(d_in, d_out) = d, the parameter count is bounded by:

2c_1c_2\binom{n}{2} = O(\log^2 d)

| Method | Complexity | LLaMA-3.2 1B Query projection params | |:---|:---|:---| | Dense | O(d²) | 4,194,304 | | LR1 (rank=1) | O(d) | 4,096 | | LR4 (rank=4) | O(4d) | 16,384 | | BH1 (Block-Hadamard) | O(d) | 32,768 | | Rotor (this paper) | O(log²d) | ≤ 896 |

Rotor vs Dense: ~4700× parameter reduction.

LLM experiments: replacing Q/K/V projections

Protocol: freeze the pretrained LLM; extract (x, y) pairs for target layers via a prompt dataset; train replacement layers with MSE loss min_θ Σ(H_θ x_i − y_i)²; replace layers sequentially.

LLaMA-3.2 1B, three-layer replacement results:

| Metric | Layers replaced | Original | LR1 | LR4 | BH1 | Rotor | |:---|:---|:---|:---|:---|:---|:---| | Wikitext2 Log-PPL ↓ | 1 | 2.575 | 2.688 | 2.658 | 2.636 | 2.629 | | | 2 | 2.575 | 3.455 | 2.729 | 2.700 | 2.717 | | | 3 | 2.575 | 4.956 | 2.880 | 2.779 | 2.818 | | C4 Log-PPL ↓ | 1 | 3.151 | 3.414 | 3.390 | 3.343 | 3.261 | | | 2 | 3.151 | 4.071 | 3.315 | 3.262 | 3.285 | | Arc Challenge ↑ | 1 | 58.37 | 50.78 | 53.84 | 54.83 | 55.31 | | | 2 | 58.37 | 50.44 | 53.39 | 54.25 | 54.50 | | HellaSwag ↑ | 1 | 41.00 | 36.17 | 38.02 | 39.10 | 39.33 |

Qwen-2.5 1.5B shows the same pattern: Rotor consistently outperforms LR4 and BH1 in one- and two-layer replacements.

Takeaway: a new paradigm for low-rank approximation

| Dimension | SVD low-rank | Rotor composition | |:---|:---|:---| | Mathematical essence | Truncated singular values, subspace projection | Exponential maps of oriented planes composed | | Parameter complexity | O(rd) | O(log²d) | | Geometric meaning | None (pure algebra) | Directly encodes rotations/reflections | | Differentiability | Direct | Requires invariant decomposition algorithms | | Empirical performance | LR4 ~1.1M params | Rotor ~1K params, matched performance | | Effect at depth | Error accumulates across layers | Small per-layer error, stable at depth |

Key insight: low-rank approximation assumes "the most important information lives in a low-dimensional subspace," while the geometric-algebra view says "a linear map decomposes into a composition of geometric rotations." The latter uses fewer parameters and preserves geometric semantics.

Part 2: Rethinking Attention

The "scalar poverty" of standard attention

Standard self-attention:

\text{Attn}(x) = \left[\text{softmax}\left(\text{mask}\left(\frac{QK^T}{\sqrt{d_k}}\right)\right)V\right]W_o

Problems:

Q·K^T compresses high-dimensional multivector relations into a scalar inner product
Direction, plane, and volume information is lost
Attention scores capture only "how similar," not "how related geometrically"

Versor's answer: Geometric Product Attention (GPA)

Core decomposition:

Q\widetilde{K} = \underbrace{\langle Q\widetilde{K}\rangle_0}_{\text{Scalar (Proximity)}} + \underbrace{\langle Q\widetilde{K}\rangle_2}_{\text{Bivector (Torque)}} + \dots

Attention scores:

\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)

Scalar part ⟨·⟩₀: distance/proximity (equivalent to traditional attention)
Bivector part ⟨·⟩₂: direction/torque (a new geometric dimension)
γ: learnable scalar balancing directional vs. proximity attention

Versor asks not just "how similar are these tokens?" but "what is their geometric relationship?"

Recursive Rotor Accumulator (RRA): O(L) sequence modeling

\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t)

Ψ_t: global state constrained to the Spin manifold (Ψ\widetilde{Ψ} = 1)
ΔR_t: a local rotor predicted at each step (via Cayley map)
Geometric product update in Cl(4,1); manifold normalization projects numerical drift back onto the Spin manifold

| Component | Standard Transformer | Versor | |:---|:---|:---| | Global relations | Self-Attention O(L²) | GPA O(L²) | | Temporal modeling | Also Attention O(L²) | RRA O(L) | | Memory | O(L²) | RRA O(1) |

Experimental validation

Chaotic 5-body gravitational dynamics (2D):

| Model | Params | MSE ↓ | Energy drift | |:---|:---|:---|:---| | Transformer (d=128) | 1.320M | 6.609 ± 6.415 | 381.1% | | Mamba | ≈0.05M | 7.4 ± 6.4 | 238.0% | | GNS | 0.026M | 5.881 ± 6.408 | 366.7% | | GATr | ≈0.1M | 8.32 ± 1.80 | 173.8% | | Versor | 0.007M | 5.210 ± 6.387 | 133.0% | | Ham-Versor | 0.044M | 4.827 ± 6.379 | 2.4% |

200× fewer parameters, 150× lower energy drift.

Topology reasoning ("Broken Snake"):

| Task | ViT/Transformer | Versor | |:---|:---|:---| | Broken Snake MCC ↑ | 0.070 | 0.993 | | Variable system size N=7 MSE ↓ | ∞ (fails) | 5.74 | | Hidden velocity inference MSE ↓ | 0.325 (GATr) | 0.003 | | OOD mass 10× Δerror | +1933.7% | −63.9% |

Zero-shot scale generalization: 0.993 vs 0.070; the Transformer collapses on OOD.

Multimodal: CIFAR-10 49.63% (3 epochs, 1.0M params, no convolutions); WikiText-2/103 perplexity 3.22.

Hardware implementation: 78× faster bit-masked kernel

| Engine | Technique | Speedup | |:---|:---|:---| | Bit-Masked Kernel | Triton/MLX, XOR-isomorphic bypass of Cayley table | 78× vs naive PyTorch | | Matrix Isomorphism | Cl(4,1) ≅ Mat(4, ℂ), GEMM-based | 65% latency reduction | | C++ RRA core | Multi-core parallel | 7.5× end-to-end latency reduction | | Final latency | — | 1.05 ms (vs Transformer 1.10 ms) |

Takeaway: a paradigm shift in attention

| Dimension | Standard Attention | Geometric Product Attention | |:---|:---|:---| | Similarity measure | Scalar dot product Q·K^T | Geometric product Q\widetilde{K} = scalar + bivector + ... | | Information dimension | 1D (how similar) | Multi-dimensional (how similar + how oriented) | | Physical equivariance | None | SE(3)-equivariant (native) | | Parameters | O(d²) | O(d_in × 32) | | Temporal modeling | O(L²) Attention | O(L) RRA + optional O(L²) GPA | | Interpretability | Black-box weights | Scalar = proximity, bivector = torque |

Part 3: A Unified View — Geometric Algebra as Deep Learning's "New Substrate"

How the two topics connect

| Topic | Core operation | Algebraic object | Paper | |:---|:---|:---|:---| | Low-rank replacement | Linear layer parameterization | Bivector → Rotor → Sandwich product | Pence et al. | | Attention redesign | Similarity computation | Geometric product → Scalar + Bivector | Hirst & Huy | | Unifying frame | Both based on | Clifford Algebra / CGA | Both |

Essential insight: traditional deep learning flattens everything into vector/matrix operations, losing the problem's geometric structure. Geometric algebra's contribution is not optimizing existing methods but redefining the representation space.

Mathematical unity

The graded structure of Clifford algebras:

Cl(p,q) = \underbrace{\langle\cdot\rangle_0}_{\text{scalar}} \oplus \underbrace{\langle\cdot\rangle_1}_{\text{vector}} \oplus \underbrace{\langle\cdot\rangle_2}_{\text{bivector}} \oplus \dots \oplus \underbrace{\langle\cdot\rangle_n}_{\text{pseudoscalar}}

Low-rank replacement: bivectors (grade 2) encode linear transformations
Attention: the full graded structure of the geometric product (scalar + bivector + ...)
Common basis: rotor = exp(bivector), geometric transformations of the Spin group

Current limitations and next steps

Pence et al. self-assessment: "It is not a drop-in replacement yet." Additional system integration is needed.

Versor's limitations:

CGA dimension fixed at Cl(4,1) (32-dim); higher-dimensional scenarios need extension
Numerical drift is controlled by manifold normalization, but large-scale training stability is unverified
NLP performance still trails (WikiText perplexity 3.22 vs Transformer 2.5–2.7)

Future directions: 1. Integrate rotor layers into LLM training pipelines (end-to-end, not frozen replacement) 2. Explore higher-dimensional Clifford algebras (Cl(5,1), Cl(3,3), etc.) across tasks 3. Hybrid GPA / standard attention architectures (the paper's "hybridization") 4. Large-scale hardware optimization (78× speedup achieved; large-scale deployment remains open)

Appendix: Key Formula Cheat Sheet

Rotor sandwich product

\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n)

Bivector → rotor (exponential map)

r = \exp(b) = \cos(\|b\|) + \frac{\sin(\|b\|)}{\|b\|} b, \quad b \wedge b = 0

Geometric Product Attention

\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)

Recursive Rotor Accumulator

\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t), \quad \Psi\widetilde{\Psi} = 1

CGA conformal embedding

X = K(x) = x + \frac{1}{2}x^2 e_\infty + e_o, \quad X_i \cdot X_j = -\frac{1}{2}\|x_i - x_j\|^2

References

1. Pence, T., Yamada, D., & Singh, V. (2025). *Composing Linear Layers from Irreducibles*. NeurIPS 2025. arXiv:2507.11688. 2. Hirst, E. & Huy, T. M. (2026). *Versor: A Geometric Sequence Architecture Enhanced Scale Generalization and Interpretability via Conformal Algebra*. arXiv:2602.10195v2. 3. Brehmer, J. et al. (2023). *Geometric Algebra Transformer*. NeurIPS 2023. https://github.com/Qualcomm-AI-research/geometric-algebra-transformer 4. Ginzberg, B. & Mavroyiakoumou, I. (2016). *SVD over Clifford Algebras*. Linear Algebra and its Applications. 5. Sadrzadeh, M. et al. (2026). *FGA for NLP*. arXiv:2604.25902. 6. Ji, S. (2026). *CliffordNet*. arXiv:2601.06793.