Geometric Algebra Restructures Deep Learning: A Dual Revolution in Low-Rank Approximation and Attention

> Research questions: (1) Can traditional SVD low-rank factorization be replaced by geometric-algebra parameterization? (2) Can the efficient rotations of rotors fundamentally redesign attention computation? > > Core papers: > - Pence et al., *Composing Linear Layers from Irreducibles*, NeurIPS 2025 (arXiv:2507.11688) > - Hirst & Huy, *Versor: A Geometric Sequence Architecture*, arXiv:2602.10195v2

---

Part 1: A New Take on Low-Rank Approximation

Problem statement

The classic SVD low-rank paradigm:

• W ≈ U_r Σ_r V_r^T with O(r(d_in + d_out)) parameters
• Essence: find an optimal r-dimensional subspace in vector space
• Limitation: r is a hyperparameter; truncating singular values discards geometric structure

Clifford algebra's answer: linear maps = compositions of bivectors

Core math (Lemma 1): any linear function can be written as a sum of sandwich products with bivector-like multivectors:

\[F(x) = \sum_{t=1}^{w} a_t x b_t\]

More specifically, the rotor sandwich product:

\[\psi_{r,s}(x) \triangleq r x s^\dagger, \quad r, s \in \text{Spin}(n)\]

where r = exp(b) and b is a simple bivector (b = u ∧ v, encoding a directed plane).

Parameterization:

• A simple bivector is parameterized by `(n,2) scalar coefficients (all planes in n-dim space)
• Rotor via exponential map: r = cos(‖b‖) + sin(‖b‖)/‖b‖ · b
• Multiple rotors aggregated via pooling: ψ(x) = σ({ψ_{r_{ij},s_{ij}}(x^{I_i})})

Differentiable factorization: from bivector to rotor

The key engineering challenge. Two algorithms are proposed:

Algorithm 1: differentiable invariant factorization — decomposes any bivector b ∈ Cl_2(n) into k = ⌊n/2⌋ mutually commuting, orthogonal simple bivectors, iteratively extracting simple components to avoid eigendecomposition instability.

Algorithm 2: GA power iteration — replaces matrix-vector products with right contractions b ⌞ v, uses closed-form projection Proj_simple(b) = σ(u ∧ v), and detects convergence via sums (not differences) to handle sign symmetry.

This makes the whole rotor parameterization end-to-end differentiable and autograd-compatible.

Parameter efficiency: the O(log² d) guarantee

Theorem 1: for a map ψ: R^{d_in} → R^{d_out} built from c_1 × c_2 rotor modules acting in Cl(n) with 2^n ≤ min(d_in, d_out) = d, the total learnable parameters satisfy:

\[2c_1c_2\binom{n}{2} = O(\log^2 d)\]

| Method | Complexity | LLaMA-3.2 1B Query projection params | |:---|:---|:---| | Dense | O(d²) | 4,194,304 | | LR1 (rank=1) | O(d) | 4,096 | | LR4 (rank=4) | O(4d) | 16,384 | | BH1 (Block-Hadamard) | O(d) | 32,768 | | Rotor (this work) | O(log²d) | ≤ 896 |

Rotor vs Dense: 4,700× fewer parameters.

LLM experiments: replacing Q/K/V projections

Protocol: freeze the pretrained LLM; extract (x, y) pairs per target layer via prompt datasets; train the replacement layer with MSE loss; replace layers sequentially.

LLaMA-3.2 1B, three-layer replacement:

| Metric | Layers | Original | LR1 | LR4 | BH1 | Rotor | |:---|:---|:---|:---|:---|:---|:---| | Wikitext2 Log-PPL ↓ | 1 | 2.575 | 2.688 | 2.658 | 2.636 | 2.629 | | | 2 | 2.575 | 3.455 | 2.729 | 2.700 | 2.717 | | | 3 | 2.575 | 4.956 | 2.880 | 2.779 | 2.818 | | C4 Log-PPL ↓ | 1 | 3.151 | 3.414 | 3.390 | 3.343 | 3.261 | | Arc Challenge ↑ | 1 | 58.37 | 50.78 | 53.84 | 54.83 | 55.31 | | HellaSwag ↑ | 1 | 41.00 | 36.17 | 38.02 | 39.10 | 39.33 |

Qwen-2.5 1.5B shows the same pattern: Rotor consistently beats LR4 and BH1 at one- and two-layer replacement.

Conclusion: a new paradigm for low-rank approximation

| Dimension | SVD low-rank | Rotor composition | |:---|:---|:---| | Math essence | Truncated singular values, subspace projection | Exponential maps of directed planes composed | | Parameter complexity | O(rd) | O(log²d) | | Geometric meaning | None (pure algebra) | Directly encodes rotations/reflections | | Differentiability | Direct | Requires invariant factorization algorithms | | Performance | LR4 ≈ 1.1M params | Rotor ≈ 1K params, matched performance | | Deep stacks | Error accumulates across layers | Small per-layer error, stable in depth |

Core insight: low-rank approximation assumes the most important information lives in a low-dimensional subspace; the geometric algebra view is that linear transformations decompose into compositions of geometric rotations. The latter uses fewer parameters and preserves geometric semantics.

---

Part 2: Rethinking Attention

The "scalar poverty" of standard attention

\[\text{Attn}(x) = \left[\text{softmax}\left(\text{mask}\left(\frac{QK^T}{\sqrt{d_k}}\right)\right)V\right]W_o\]

Problems:

• Q·K^T compresses high-dimensional multivector relations into a scalar inner product
• Loses direction, plane, and volume information
• Attention scores have only a "how similar" dimension, no "how does it rotate"

Versor's answer: Geometric Product Attention (GPA)

\[Q\widetilde{K} = \underbrace{\langle Q\widetilde{K}\rangle_0}_{\text{Scalar (proximity)}} + \underbrace{\langle Q\widetilde{K}\rangle_2}_{\text{Bivector (torque)}} + \dots\]
\[\alpha_{ij} = \text{softmax}\left(\frac{\langle Q_i\widetilde{K}_j\rangle_0 + \gamma \|\langle Q_i\widetilde{K}_j\rangle_2\|}{\sqrt{d_{in}}}\right)\]
• Scalar part ⟨·⟩₀: distance/proximity (the traditional attention equivalent)
• Bivector part ⟨·⟩₂: direction/torque (a new geometric dimension)
• γ: learnable scalar balancing directional vs proximity attention

Versor asks not only "how similar are these tokens" but "what is their geometric relationship."

Recursive Rotor Accumulator (RRA): O(L) sequence modeling

\[\Psi_{t+1} = \text{Normalize}(\Delta R_t \Psi_t)\]
• Ψ_t: global state constrained to the Spin manifold (ΨΨ̃ = 1)
• ΔR_t: a local rotor predicted per step (via Cayley map)
• Geometric product update in Cl(4,1); manifold normalization projects numerical drift back onto the Spin manifold

| Component | Standard Transformer | Versor | |:---|:---|:---| | Global relations | Self-Attention O(L²) | GPA O(L²) | | Temporal modeling | Attention O(L²) | RRA O(L) | | Memory | O(L²) | RRA O(1) |

Experiments: overwhelming advantages

Chaotic N-body dynamics (5 bodies, 2D gravity):

| Model | Params | MSE ↓ | Energy drift | |:---|:---|:---|:---| | Transformer (d=128) | 1.320M | 6.609 ± 6.415 | 381.1% | | Mamba | ≈0.05M | 7.4 ± 6.4 | 238.0% | | GNS | 0.026M | 5.881 ± 6.408 | 366.7% | | GATr | ≈0.1M | 8.32 ± 1.80 | 173.8% | | Versor | 0.007M | 5.210 ± 6.387 | 133.0% | | Ham-Versor | 0.044M | 4.827 ± 6.379 | 2.4% |

200× fewer parameters, 150× lower energy drift.

Topological reasoning ("Broken Snake"):

| Task | ViT/Transformer | Versor | |:---|:---|:---| | Broken Snake MCC ↑ | 0.070 | 0.993 | | Variable size N=7 MSE ↓ | ∞ (fails) | 5.74 | | Hidden velocity inference MSE ↓ | 0.325 (GATr) | 0.003 | | OOD 10× mass Δ error | +1933.7% | −63.9% |

Zero-shot scale generalization: 0.993 vs 0.070 — Transformers collapse OOD.

Multimodal: CIFAR-10: 49.63% (3 epochs, 1.0M params, no convolutions); WikiText-2/103: perplexity 3.22.

Hardware: a 78× speedup via bit-masked kernels

| Engine | Technique | Speedup | |:---|:---|:---| | Bit-Masked Kernel | Triton/MLX, XOR isomorphism bypassing Cayley tables | 78× vs naive PyTorch | | Matrix Isomorphism | Cl(4,1) ≅ Mat(4, ℂ), GEMM-based | 65% latency reduction | | C++ RRA core | Multi-core parallel | 7.5× end-to-end latency reduction | | Final latency | — | 1.05 ms (vs Transformer 1.10 ms) |

Conclusion: a paradigm shift for attention

| Dimension | Standard attention | Geometric Product Attention | |:---|:---|:---| | Similarity measure | Scalar dot product Q·K^T | Geometric product: scalar + bivector + ... | | Information dimension | 1D (similarity) | Multi-dim (similarity + rotation) | | Physical equivariance | None | Native SE(3)-equivariance | | Parameters | O(d²) | O(d_in × 32) | | Sequence modeling | O(L²) | O(L) RRA + optional O(L²) GPA | | Interpretability | Black-box weights | Scalar = proximity, bivector = torque |

---

Part 3: A Unified View — Geometric Algebra as Deep Learning's "New Substrate"

| Topic | Core operation | Algebraic object | Paper | |:---|:---|:---|:---| | Low-rank replacement | Linear layer parameterization | Bivector → Rotor → Sandwich product | Pence et al. | | Attention redesign | Similarity computation | Geometric product → scalar + bivector | Hirst & Huy | | Unifying framework | Both rely on | Clifford Algebra / CGA | Both |

Essential insight: traditional deep learning flattens everything into vector/matrix operations, discarding the problem's geometric structure. Geometric algebra's contribution is not "optimizing existing methods" but "redefining the representation space."

Both approaches draw on the graded structure of Clifford algebra:

\[Cl(p,q) = \langle\cdot\rangle_0 \oplus \langle\cdot\rangle_1 \oplus \langle\cdot\rangle_2 \oplus \dots \oplus \langle\cdot\rangle_n\]
• Low-rank approximation: bivectors (grade 2) encode linear transformations
• Attention: the full graded structure of the geometric product
• Common foundation: rotor = exp(bivector), geometric transformations of the Spin group

Limitations and next steps

Pence et al. self-assess: "It is not a drop-in replacement yet" — additional systems integration is required.

Versor's limitations:

• CGA dimension fixed at Cl(4,1) (32-dim); higher-dimensional extensions needed
• Numerical drift is controlled by manifold normalization, but large-scale training stability is unverified
• NLP performance still lags (WikiText perplexity 3.22 vs Transformers' 2.5–2.7)

Future directions: 1. Integrate rotor layers into LLM training pipelines (end-to-end, not frozen replacement) 2. Explore higher-dimensional Clifford algebras (Cl(5,1), Cl(3,3), etc.) across tasks 3. Hybrid GPA + standard attention architectures ("hybridization" per the paper) 4. Large-scale hardware optimization (78× speedup already achieved; deployment headroom remains)

---

Appendix: Key Formulas

Rotor sandwich product

Bivector → rotor (exponential map)

Geometric Product Attention

Recursive Rotor Accumulator

CGA conformal embedding

---

References

1. Pence, T., Yamada, D., & Singh, V. (2025). *Composing Linear Layers from Irreducibles*. NeurIPS 2025. arXiv:2507.11688. 2. Hirst, E. & Huy, T. M. (2026). *Versor: A Geometric Sequence Architecture Enhanced Scale Generalization and Interpretability via Conformal Algebra*. arXiv:2602.10195v2. 3. Brehmer, J. et al. (2023). *Geometric Algebra Transformer*. NeurIPS 2023. https://github.com/Qualcomm-AI-research/geometric-algebra-transformer 4. Ginzberg, B. & Mavroyiakoumou, I. (2016). *SVD over Clifford Algebras*. Linear Algebra and its Applications. 5. Sadrzadeh, M. et al. (2026). *FGA for NLP*. arXiv:2604.25902. 6. Ji, S. (2026). *CliffordNet*. arXiv:2601.06793.