The Problem: Is SVD Low-Rank Decomposition Wrong?
Since AlexNet in 2012, the dominant parameterization in deep learning has been matrix multiplication—dense matrices, low-rank approximations, block-diagonal matrices. SVD became the standard tool for compressing neural networks: factor a large matrix W into UΣVᵀ, discard small singular values, and reduce parameters from d² to 2dr.
But this paradigm carries a hidden assumption: that the best parameterization of a linear transformation is a rank-constrained matrix factorization.
A NeurIPS 2025 paper raises a fundamental challenge: what if the natural decomposition of a linear transformation is not SVD, but rotor composition from geometric algebra?
---
Direction One: A New Take on Low-Rank Approximation
Where Does SVD Fall Short?
Geometrically, SVD is: rotate → scale → rotate. U and V handle rotation, Σ handles scaling.
But this decomposition is global—it acts on the entire d×d space. When d = 4096 (a common LLM dimension), U and V each need d² parameters. Low-rank approximation compresses via truncation, but the discarded singular values are not unimportant information—only unimportant "in the Frobenius norm sense."
More critically, SVD is static. It treats the matrix as a black box of numbers, ignoring any geometric structure those numbers may encode.
The Clifford Algebra View: Linear Maps = Compositions of Oriented Planes
The core insight of Clifford algebra (geometric algebra): a linear transformation can be decomposed as a composition of bivectors (oriented planes).
Any rotation in 3D can be expressed as a turn about some axis—and that axis is perpendicular to a plane. That plane is a bivector. Composition of multiple rotations = products of exponentials of multiple bivectors.
Core result of "Composing Linear Layers from Irreducibles" (NeurIPS 2025):
> Linear layers can be represented as compositions of bivectors (geometric objects encoding oriented planes), acting via the rotor sandwich product on local subspaces of input multivectors. This requires only O(log²d) parameters, versus O(d²) for dense matrices.
Implementation:
- Embed input vectors into the multivector space of a Clifford algebra
- Apply the rotor sandwich product ψᵣ,ₛ(x) ≜ rxs†, with r, s ∈ Spin(n)
- Each rotor is parameterized by a small number of bivector coefficients
- Map bivectors to rotors via a differentiable invariant factorization algorithm (iterative extraction inspired by Krylov subspaces)
- Original dense layers: 1–4M parameters
- Rotor layers: ~1000 parameters (width=2, depth=3)
- Performance: matches block-Hadamard and low-rank approximations (LR1 needs 3–5× more parameters)
- Geometric Product Attention (GPA) — geometric product replaces the dot product
- RecursiveRotorAccumulator — rotor-based sequence pooling
- Conformal lift — 4D points lifted into Cl(4,1) multivectors
- Standard attention is mostly linear aggregation after softmax and needs a heavy FFN (ratio=4) for feature transformation capacity
- The Clifford geometric product contains built-in second-order multiplicative terms (bivector u∧v and scalar u⊙v)
- Combined with nonlinear GGR gating, the interaction layer is itself a powerful function approximator
- FFNs do dense, indiscriminate channel mixing via large matrix multiplications
- The Shifted Geometric Product performs structured mixing based on the cyclic topology of the feature space
- Geometry-constrained mixing is more sample-efficient than brute-force connectivity
- BERTology shows different attention heads specialize in different linguistic functions (syntactic dependencies, coreference, semantic roles, positional/discourse features)
- These specializations are structurally equivalent to Clifford algebra's grade decomposition: each head operates in a learned projected subspace
- Multi-head attention combines information from subspaces of different directions, just as Clifford algebra decomposes by grade and direction
- Current transformers: contextual modulation via learned linear projections (Q, K, V matrices)
- FGA proposal: contextual modulation via the rotor sandwich—a grade-preserving, invertible transformation
- Rotors preserve norm and grade; projections do not
- Rotors are invertible; projections are not
- SVD: O(dr) parameters, rank-constrained, static factorization
- Rotor: O(log²d) parameters, geometry-constrained, dynamic composition
- Key insight: full-rank transforms aren't needed; small orthogonal blocks suffice on low-rank manifolds
- Standard: dot product → scalar → softmax → linear aggregation → FFN reconstruction
- GA-based: geometric product → multivector → grade-aware gating → internalized nonlinearity
- Key insight: preserving bivector components = preserving relational directional information, with no need for post-hoc FFN compensation
- RNNs: hidden states updated via gated recurrence
- Transformers: global aggregation via attention
- Versor: state updated via rotor evolution, with the geometric product as a native computational primitive
- LLM efficiency researchers (KV cache compression, parameter-efficient fine-tuning)
- Geometric deep learning practitioners (3D vision, physics simulation, molecular dynamics)
- Theorists interested in whether neural networks can be more structured
- Explorers seeking architectures beyond the transformer
- Scenarios requiring immediate production deployment (system-level optimization of GA architectures is early)
- Tasks dominated by non-geometric/non-structured data (GA's advantages are most pronounced on geometric data)
- Projects demanding minimal training infrastructure (requires torch_ga or custom kernels)
- GATr: https://github.com/Qualcomm-AI-research/geometric-algebra-transformer (Brehmer et al., NeurIPS 2023)
- Composing Linear Layers from Irreducibles: https://arxiv.org/abs/2507.11688 (Pence et al., NeurIPS 2025)
- RotorQuant: https://github.com/scrya-com/rotorquant (scrya, 2026)
- CliffordNet: https://arxiv.org/abs/2601.06793 (Ji, 2026)
- Versor: https://github.com/VersorAI/Versor (Huy & Hirst, 2026)
- FGA for NLP: https://arxiv.org/abs/2604.25902 (Sadrzadeh et al., 2026)
- SVD over Clifford Algebras: Ginzberg & Mavroyiakoumou (Linear Algebra and its Applications, 2016)
Results on LLaMA-3.2 1B Q/K/V projections:
RotorQuant: From Theory to Production
RotorQuant is the engineering-focused version of rotor-based parameterization, targeting KV cache compression.
Conventional approaches (e.g., TurboQuant) apply a dense d×d random orthogonal rotation to decorrelate KV cache vectors before quantization—costing 16,384 FMAs at d=128.
RotorQuant's insight: > A full-rank d×d transform isn't needed for decorrelation. Small orthogonal blocks suffice, because real attention vectors live on low-rank manifolds.
Implementation: 1. Split the d-dimensional KV vector into chunks of 3 (d=128 → 43 groups) 2. Embed each group as a Cl(3,0) multivector (8 components) 3. Apply the rotor sandwich product RxR̃ per group to decorrelate 4. Each rotor has only 4 nonzero components (scalar + 3 bivectors)
Comparison:
| Method | FMAs (d=128) | Params | PPL | Status | |--------|--------------|--------|-----|--------| | TurboQuant | 16,384 | 16,384 | baseline | production | | RotorQuant | ~2,400 | 372 | matched | research | | IsoQuant (quaternion 4D) | 512 | 128 | better | production (llama.cpp) | | PlanarQuant (Givens 2D) | 256 | 128 | better | production (llama.cpp) |
An interesting finding: the richer the algebra (RotorQuant > IsoQuant > PlanarQuant), the worse the practical result. Simpler rotations work better—block-diagonal rotations preserve the directional structure of KV cache vectors more effectively than global WHT scrambling.
This is engineering pragmatism winning over algebraic purism: Clifford algebra provides the insight, but the effective implementation is the simplified version.
---
Direction Two: Rethinking Attention
The "Scalar Poverty" of Standard Attention
Standard transformer attention:
> attention(Q, K, V) = softmax(QKᵀ/√d) V
The interaction between Q and K is compressed into a scalar dot product. Complex high-dimensional relations between d-dimensional vectors collapse into a single number.
CliffordNet's critique is sharp: > This operation is inherently lossy: it discards the directional and structural information (bivector components) inherent in the feature space. Transformers must therefore rely on heavy, parameter-inefficient FFNs to reconstruct and process these lost feature dimensions.
In other words: attention crushes high-dimensional geometric relations into scalars, then FFNs spend enormous parameters inflating them back. That's not efficiency—it's waste.
GATr's Answer: Multivector Attention
Qualcomm AI Research's GATr (NeurIPS 2023) was the first full transformer built in PGA (Projective Geometric Algebra).
Its attention is not scalar but multivector-based, combining three sources of information: 1. Multivector inner product — the scalar part, analogous to standard dot products 2. Distance-aware nonlinear features — geometric features based on position/distance 3. Auxiliary scalars — extra scalar channels for non-geometric information
Key design: all internal states are multivectors (scalar + vector + bivector + ...), attention preserves grade structure, and gated nonlinearities mediate cross-grade interactions.
Results: on geometric tasks like n-body simulation and vessel segmentation, GATr matches or exceeds SE(3)-Transformer and SEGNN with 1/10 the parameters.
Versor: Rotor-Based Sequence Modeling
Versor (Huy & Hirst, 2026) goes further: a sequence model built in Conformal GA (Cl(4,1)), where temporal state updates are performed via rotor-based transformations.
Core design:
Versor outperforms transformer, graph, and GA-based baselines on multiple benchmarks with significantly fewer parameters.
CliffordNet: The No-FFN Lesson
CliffordNet's most counterintuitive finding: the FFN isn't needed.
On CIFAR-100, FFN-free CliffordNet-Nano reaches 76.41%, beating ShuffleNetV2's 74.60%. FFN-free CliffordNet-Fast hits 77.63%, far above MobileNetV2's 70.90%.
Why?
Algebraic density:
Structured vs. brute-force mixing:
FGA: A Theoretical Framework for Attention as Geometric Interaction
Sadrzadeh et al. (2026), "Toward a Functional Geometric Algebra for Natural Language Semantics," make an even bolder claim:
> The bivector components discarded by current attention are exactly the relational directional information needed for compositional semantics.
Multi-head attention = implicit subspace selection:
Rotor vs. projection:
Whether rotor-based contextual modulation yields better compositional generalization than projection-based approaches on sense disambiguation tasks (e.g., "Romeo and Juliet" vs. "Alfa Romeo") remains an open experimental question.
---
The Bigger Picture: A New Paradigm for Geometric Deep Learning
Three Evolutionary Paths
1. Low-rank replacement (from SVD to rotors)
2. Attention restructuring (from scalars to multivectors)
3. Sequence modeling (from RNN/Transformer to rotor evolution)
Challenges and Limitations
1. Compute efficiency. Clifford operations cost more than standard matrix multiplication. CliffordNet admits: "Baseline CNNs benefit from mature cuDNN kernel optimizations." GA operations need custom kernels to reach full potential.
2. Dimension constraints. Rotor-based layers require power-of-two dimensions (Clifford algebra dimension = 2ⁿ). Arbitrary din/dout requires composing multiple rotor-sandwich modules, adding architectural complexity.
3. Interpretability barrier. Multivector grade decomposition (scalar + vector + bivector + trivector) offers a new interpretability dimension, but raises the comprehension bar—less intuitive than attention weights.
4. Training stability. The rotor sandwich product is grade-preserving, requiring extra permutations and normalization to enable cross-grade interaction. The paper reports normalization is critical for training stability.
---
Who Should Care?
Worth following:
Not yet for:
---
Conclusion
GATr and its successors (Versor, CliffordNet, RotorQuant, FGA) are building a unified framework: re-describing the fundamental operations of deep learning in the language of geometric algebra.
Three core insights:
1. Low-rank approximation can be replaced by rotor composition — not by truncating singular values, but by composing rotations of oriented planes, dropping parameters from O(dr) to O(log²d).
2. Attention's scalar poverty can be cured by multivector richness — rather than crushing high-dimensional relations into scalars and re-inflating them, the geometric product preserves the full algebraic structure.
3. FFNs can be replaced by internalized nonlinearity — the geometric product's second-order terms (bivector + scalar) provide sufficient transformation capacity on their own, without an external MLP.
These ideas remain at the feasibility-validation stage—per the NeurIPS 2025 paper: "It is not a drop-in replacement yet." But the direction is clear: the next paradigm shift in deep learning may come not from bigger models, but from more structured representations.
After all, if nature describes physics with geometric algebra (from Maxwell's equations to the Dirac equation), perhaps neural networks should too.
---
Key sources