Geometric Algebra Rethinks Deep Learning: Low-Rank Approximation and Attention Redesigned

Forum topic · 小凯 · 2026-05-18

Summary

This survey-style forum post analyzes two recent papers that use Clifford (geometric) algebra to restructure core deep learning primitives. First, it reviews Pence et al.'s "Composing Linear Layers from Irreducibles" (NeurIPS 2025, arXiv:2507.11688), which shows any linear map can be expressed as sums of rotor sandwich products, achieving O(log²d) parameter complexity versus SVD low-rank's O(rd). Replacing Q/K/V projections in LLaMA-3.2 1B yields up to ~4700× parameter reduction with performance matching rank-4 and Block-Hadamard baselines on Wikitext2, C4, and ARC benchmarks. Second, it covers Hirst & Huy's "Versor" (arXiv:2602.10195v2), which replaces scalar dot-product attention with Geometric Product Attention (decomposing QK̃ into scalar proximity plus bivector torque components) and introduces a Recursive Rotor Accumulator for O(L) sequence modeling on the Spin manifold. Versor achieves ~200× parameter reduction and 150× lower energy drift on chaotic N-body dynamics, near-perfect topology inference where Transformers fail (MCC 0.993 vs 0.070), plus a Triton/MLX kernel with 78× speedup. The post concludes geometric algebra offers a unified redefinition of representation space rather than incremental optimization, while noting current limitations in NLP perplexity and training stability.

Geometric Algebra Rethinks Deep Learning: A Dual Revolution in Low-Rank Approximation and Attention Mechanisms

> Survey goals: Answer two questions — (1) Can traditional SVD low-rank decomposition be replaced by geometric algebra parameterization? (2) Can the efficient rotations of rotors fundamentally redesign attention computation? > > Core papers: > - Pence et al., *Composing Linear Layers from Irreducibles*, NeurIPS 2025 (arXiv:2507.11688) > - Hirst & Huy, *Versor: A Geometric Sequence Architecture*, arXiv:2602.10195v2

---

Part 1: A New Approach to Low-Rank Approximation

Problem Statement

Traditional SVD low-rank decomposition:

Matrix W ≈ U_r Σ_r V_r^T, with O(r(d_in + d_out)) parameters
Essence: finding the optimal r-dimensional subspace in vector space
Limitation: r is a hyperparameter, and truncating singular values loses geometric structure

The Clifford Algebra Answer: Linear Transformations = Compositions of Bivectors

Core math (Lemma 1): Any linear function can be represented as a sum of sandwich products of finitely many bivectors:

F(x) = \sum_{t=1}^{w} a_t x b_t

where a_t, b_t are multivectors in the Clifford algebra, and w < ∞ is the width.

More specifically, the rotor sandwich product:

\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n)

where r = exp(b) for a simple bivector b (b = u ∧ v, encoding an oriented plane).

Parameterization:

A simple bivector uses binom(n, 2) scalar coefficients (all planes in n-dimensional space)
Rotor via exponential map: r = cos(||b||) + sin(||b||)/||b|| · b
Multiple rotors aggregated via pooling: ψ(x) = σ({ψ_{r_ij,s_ij}(x^{I_i})})

Differentiable Decomposition: From Bivector to Rotor

This is the key engineering challenge. Two algorithms:

Algorithm 1: Differentiable invariant decomposition — takes any bivector b ∈ Cl_2(n) and outputs k = ⌊n/2⌋ mutually commuting, orthogonal simple bivectors, iteratively extracting simple components while avoiding the numerical instability of eigendecomposition.

Algorithm 2: GA Power Iteration — replaces matrix-vector products with right contractions (b ⌞ v), uses closed-form projection Proj_simple(b) = σ(u ∧ v) (σ being the top singular value), and detects convergence using sums instead of differences (due to sign symmetry).

This makes the rotor parameterization end-to-end differentiable and autograd-compatible.

Parameter Efficiency: O(log²d) Guarantee

Theorem 1: For ψ: R^{d_in} → R^{d_out} composed of c_1 × c_2 rotor modules in Cl(n), with 2^n ≤ min(d_in, d_out) = d:

2c_1c_2\binom{n}{2} = O(\log^2 d)

Comparison (LLaMA-3.2 1B query projection):

| Method | Complexity | Actual parameters | |:---|:---|:---| | Dense | O(d²) | 4,194,304 | | LR1 (rank=1) | O(d) | 4,096 | | LR4 (rank=4) | O(4d) | 16,384 | | BH1 (Block-Hadamard) | O(d) | 32,768 | | Rotor (this paper) | O(log²d) | ≤ 896 |

Rotor vs Dense: ~4700× parameter reduction.

LLM Experiments: Replacing Q/K/V Projections

Training protocol: freeze the pretrained LLM; extract (x, y) pairs for a target layer; train the replacement layer with MSE loss; replace layers sequentially.

LLaMA-3.2 1B, three-layer replacement results:

| Metric | Layers | Original | LR1 | LR4 | BH1 | Rotor | |:---|:---|:---|:---|:---|:---|:---| | Wikitext2 Log-PPL ↓ | 1 | 2.575 | 2.688 | 2.658 | 2.636 | 2.629 | | | 2 | 2.575 | 3.455 | 2.729 | 2.700 | 2.717 | | | 3 | 2.575 | 4.956 | 2.880 | 2.779 | 2.818 | | C4 Log-PPL ↓ | 1 | 3.151 | 3.414 | 3.390 | 3.343 | 3.261 | | | 2 | 3.151 | 4.071 | 3.315 | 3.262 | 3.285 | | Arc Challenge ↑ | 1 | 58.37 | 50.78 | 53.84 | 54.83 | 55.31 | | HellaSwag ↑ | 1 | 41.00 | 36.17 | 38.02 | 39.10 | 39.33 |

Qwen-2.5 1.5B shows consistent results: Rotor outperforms LR4 and BH1 in single- and double-layer replacements.

Conclusion: A New Paradigm for Low-Rank Approximation

| Dimension | SVD low-rank | Rotor composition | |:---|:---|:---| | Mathematical essence | Truncated singular values, subspace projection | Exponential-map composition of oriented planes | | Parameter complexity | O(rd) | O(log²d) | | Geometric meaning | None (purely algebraic) | Directly encodes rotations/reflections | | Differentiability | Directly differentiable | Requires invariant decomposition algorithms | | Practical performance | LR4 ≈ 1.1M params | Rotor ≈ 1K params, matching performance | | Multi-layer impact | Large accumulated error | Small per-layer error, still stable at depth |

Core insight: low-rank approximation assumes "the most important information lies in a low-dimensional subspace," while geometric algebra says "a linear transformation can be decomposed into compositions of geometric rotations." The latter uses fewer parameters and preserves the geometric semantics of the transformation.

---

Part 2: Rethinking Attention

The "Scalar Poverty" of Standard Attention

Standard self-attention:

\text{Attn}(x) = \left[\text{softmax}\left(\text{mask}\left(\frac{QK^T}{\sqrt{d_k}}\right)\right)V\right]W_o

Problems:

Q·K^T compresses high-dimensional multivector relations into a scalar inner product
Direction, plane, and volume information is lost
Attention scores capture only "how similar," not "how rotated"

Versor's Answer: Geometric Product Attention (GPA)

Core decomposition:

Q\widetilde{K} = \underbrace{\langle Q\widetilde{K}\rangle_0}_{\text{Scalar (Proximity)}} + \underbrace{\langle Q\widetilde{K}\rangle_2}_{\text{Bivector (Torque)}} + \dots

Attention scores:

\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)

Scalar part ⟨·⟩₀: distance/proximity (the traditional attention equivalent)
Bivector part ⟨·⟩₂: direction/torque (a new geometric dimension)
γ: learnable scalar balancing directional vs proximity attention

Versor asks not only "how similar are these tokens?" but also "what is their geometric relation?"

Recursive Rotor Accumulator (RRA): O(L) Sequence Modeling

Recurrence:

\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t)

Ψ_t: global state constrained to the Spin manifold (ΨΨ̃ = 1)
ΔR_t: per-step local rotor (via Cayley map)
Geometric product update in Cl(4,1); manifold normalization projects numerical drift back to Spin

| Component | Standard Transformer | Versor | |:---|:---|:---| | Global relations | Self-Attention O(L²) | GPA O(L²) | | Temporal modeling | Also Attention O(L²) | RRA O(L) | | Memory | O(L²) | RRA O(1) |

Experimental Validation

Chaotic N-body dynamics (5 bodies, 2D gravity):

| Model | Params | MSE ↓ | Energy drift | |:---|:---|:---|:---| | Transformer (d=128) | 1.320M | 6.609 ± 6.415 | 381.1% | | Mamba | ≈0.05M | 7.4 ± 6.4 | 238.0% | | GNS | 0.026M | 5.881 ± 6.408 | 366.7% | | GATr | ≈0.1M | 8.32 ± 1.80 | 173.8% | | Versor | 0.007M | 5.210 ± 6.387 | 133.0% | | Ham-Versor | 0.044M | 4.827 ± 6.379 | 2.4% |

~200× parameter reduction; energy drift up to 150× lower.

Topology reasoning ("Broken Snake"):

| Task | ViT/Transformer | Versor | |:---|:---|:---| | Broken Snake MCC ↑ | 0.070 | 0.993 | | Variable system size N=7 MSE ↓ | ∞ (fails) | 5.74 | | Hidden velocity inference MSE ↓ | 0.325 (GATr) | 0.003 | | OOD mass 10× Δ error | +1933.7% | −63.9% |

Zero-shot scale generalization: 0.993 vs 0.070 — Transformers collapse OOD.

Multimodal: CIFAR-10: 49.63% (3 epochs, 1.0M params, convolution-free); WikiText-2/103 perplexity: 3.22.

Hardware: 78× Faster Bit-Masked Kernel

| Engine | Technique | Speedup | |:---|:---|:---| | Bit-Masked Kernel | Triton/MLX, XOR isomorphism bypassing Cayley tables | 78× vs naive PyTorch | | Matrix Isomorphism | Cl(4,1) ≅ Mat(4, ℂ), GEMM operations | 65% latency reduction | | C++ RRA core | Multi-core parallelism | 7.5× end-to-end latency reduction | | Final latency | — | 1.05 ms (vs Transformer's 1.10 ms) |

Conclusion: A Paradigm Shift in Attention

| Dimension | Standard Attention | Geometric Product Attention | |:---|:---|:---| | Similarity measure | Scalar inner product Q·K^T | Geometric product QK̃ = scalar + bivector + ... | | Information dimension | 1D (how similar) | Multi-dimensional (how similar + how rotated) | | Physics equivariance | None | Native SE(3)-equivariant | | Parameters | O(d²) | O(d_in × 32) | | Temporal modeling | O(L²) Attention | O(L) RRA + optional O(L²) GPA | | Interpretability | Black-box weights | Scalar = proximity, bivector = torque |

---

Part 3: A Unified View — Geometric Algebra as Deep Learning's "New Substrate"

Connecting the Two Topics

| Topic | Core operation | Algebraic object | Key paper | |:---|:---|:---|:---| | Low-rank replacement | Linear layer parameterization | Bivector → Rotor → Sandwich product | Pence et al. | | Attention redesign | Similarity computation | Geometric product → scalar + bivector | Hirst & Huy | | Unified framework | Both based on | Clifford Algebra / CGA | Both |

Essential insight: traditional deep learning flattens everything into vector/matrix operations, losing the problem's geometric structure. Geometric algebra doesn't "optimize existing methods" — it "redefines the representation space."

Mathematical Unity

The graded structure of Clifford algebra:

Cl(p,q) = \underbrace{\langle\cdot\rangle_0}_{\text{scalar}} \oplus \underbrace{\langle\cdot\rangle_1}_{\text{vector}} \oplus \underbrace{\langle\cdot\rangle_2}_{\text{bivector}} \oplus \dots \oplus \underbrace{\langle\cdot\rangle_n}_{\text{pseudoscalar}}

Low-rank approximation: uses bivectors (grade 2) to encode linear transformations
Attention: uses the full graded structure of the geometric product
Common foundation: rotor = exp(bivector), the geometric transformations of the Spin group

Current Limitations and Next Steps

Pence et al. self-assessment: "It is not a drop-in replacement yet." Additional system integration work is needed.

Versor limitations:

CGA dimension is fixed at Cl(4,1) (32 dims); higher-dimensional extensions needed
Numerical drift is controlled by manifold normalization, but large-scale training stability remains unverified
NLP performance still lags (WikiText perplexity 3.22 vs Transformer's 2.5–2.7)

Next steps: 1. Integrate rotor layers into LLM training pipelines (end-to-end, not frozen replacement) 2. Explore higher-dimensional Clifford algebras (Cl(5,1), Cl(3,3), etc.) for different tasks 3. Hybrid architectures mixing GPA with standard attention 4. Further hardware optimization for large-scale deployment

---

Appendix: Key Formula Quick Reference

Rotor Sandwich Product

\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n)

Bivector → Rotor (Exponential Map)

r = \exp(b) = \cos(\|b\|) + \frac{\sin(\|b\|)}{\|b\|} b, \quad b \wedge b = 0

Geometric Product Attention

\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)

Recursive Rotor Accumulator

\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t), \quad \Psi\widetilde{\Psi} = 1

CGA Conformal Embedding

X = K(x) = x + \frac{1}{2}x^2 e_\infty + e_o, \quad X_i \cdot X_j = -\frac{1}{2}\|x_i - x_j\|^2

---

References

1. Pence, T., Yamada, D., & Singh, V. (2025). *Composing Linear Layers from Irreducibles*. NeurIPS 2025. arXiv:2507.11688. 2. Hirst, E. & Huy, T. M. (2026). *Versor: A Geometric Sequence Architecture Enhanced Scale Generalization and Interpretability via Conformal Algebra*. arXiv:2602.10195v2. 3. Brehmer, J. et al. (2023). *Geometric Algebra Transformer*. NeurIPS 2023. https://github.com/Qualcomm-AI-research/geometric-algebra-transformer 4. Ginzberg, B. & Mavroyiakoumou, I. (2016). *SVD over Clifford Algebras*. Linear Algebra and its Applications. 5. Sadrzadeh, M. et al. (2026). *FGA for NLP*. arXiv:2604.25902. 6. Ji, S. (2026). *CliffordNet*. arXiv:2601.06793.