A Strange Intuition
Imagine holding a Rubik's cube.
The traditional approach says: describe its state by recording every sticker—54 colors, a "list-style" description. Red here, blue there, yellow on the back...
But there's another way: record what rotations the cube went through. Starting from the solved state, rotate 30° around the x-axis, 45° around a diagonal, then turn the top layer 90°. This is a "transformation-style" description—it records not the result but the process.
Deep learning's low-rank approximation has long been stuck in the first mode of thinking.
SVD: A Great List
SVD (Singular Value Decomposition) is a mathematical marvel. Any matrix \(A\) decomposes into:
The singular values on the diagonal of \(\Sigma\) tell you in which directions a transformation stretches the most. But singular values have a problem: they are scalars.
Think of a stretchy fabric: SVD flattens it into a plane, marks how much each cell is stretched, then sews it back. The geometry—curves, twists, rotations—gets reduced to pure numbers. This works well in many cases, but what if your data carries geometric meaning? 3D point clouds, molecular conformations, rigid body motion...
Geometric Algebra: Recovering Lost Structure
In the 19th century, William Clifford invented an algebra system that treats vectors, rotations, and scaling uniformly. In Geometric Algebra, a multivector can simultaneously contain:
- Scalar: a pure number
- Vector: a directed quantity
- Bivector: representing a plane of rotation (like a flag—the pole is the axis, the flag's area is the angle)
- Trivector: volume element
- ...
- better continual learning
- smoother latent space structure
- potentially better generalization
In 3D space, these combine into a 16-dimensional space (in Projective Geometric Algebra). Strange, yes—but rotations are native to this space.
Rotor: The DNA of Rotation
In Geometric Algebra, rotations are represented by a Rotor:
where \(B\) is a bivector encoding the rotation plane and angle. Crucially, rotors compose: the product of two rotors describes "first this rotation, then that one." A chain of rotors encodes a complex motion like a strand of DNA.
That's the core idea:
> Instead of approximating a linear transformation with scalar singular values, encode it with a chain of rotors.
The Sandwich Product
A rotor acts on a vector via the "sandwich product":
As long as \(R\) is a unit rotor, this operation is guaranteed to be an orthogonal transformation—length-preserving, angle-preserving, a pure geometric rotation.
Compare with SVD: linearly interpolating between two SVD decompositions usually yields an invalid transformation—singular values can go negative, geometry breaks. But interpolating between rotors (SLERP, spherical linear interpolation) always produces a valid rotation.
Experiments: Theory vs. Reality
Experiment 1: Rotation Transform
Task: learn a 3D rotation.
| Method | Params | Test Error | Train Time | |--------|--------|------------|------------| | Standard MLP | 115 | 2.52 | 0.028s | | SVD low-rank (rank=2) | 91 | 2.40 | 0.031s | | Rotor cascade (2 rotors) | 411 | 2.52 | 0.125s |
A sobering result: the rotor method is parameter-inefficient. Why? Rotors live in a 16-dimensional PGA space; the input must first be projected there via a projection layer costing \(16 \times d_{in}\) parameters—huge overhead for small dimensions.
Experiment 2: Function Fitting
Task: fit a random linear function. Similar outcome: comparable accuracy, but more parameters and time.
Experiment 3: Parameter Efficiency
| Config | SVD Params | Rotor Params | Ratio | |--------|------------|--------------|-------| | (10, 32, 5, rank=2) | 86 | 760 | 8.8× | | (10, 32, 5, rank=8) | 344 | 784 | 2.3× | | (100, 128, 50, rank=8) | 1832 | 3952 | 2.2× |
Notably, the relative overhead of rotors shrinks with dimension: the projection cost is fixed while SVD parameters grow linearly.
Why Not Give Up?
Despite the current disadvantage, three reasons to continue:
1. Geometric Consistency
Singular values are just numbers. Rotors correspond to real rotations: "this transform is approximated by 3 rotors" means "decomposable into 3 simple geometric motions." For interpretable transformations (robot motion planning, physics simulation), this could be a major advantage.
2. Interpolation and Generalization
Straight-line interpolation in parameter space between SVD networks is usually meaningless. Rotor interpolation stays geometrically valid, suggesting:
3. Equivariance
Rotor decomposition naturally preserves E(3) equivariance (rotation, translation, reflection). For tasks requiring this symmetry—molecular property prediction, physical simulation—rotors may be more elegant than constrained SVD.
Next Step: The Return of Quaternions
The bottleneck is the 16-dim PGA projection overhead. But for pure rotations, a rotor is equivalent to a quaternion:
A quaternion needs only 4 parameters instead of 16. Next step: represent rotors directly as quaternions, eliminating the PGA projection overhead—potentially matching SVD efficiency while keeping the geometric advantages.
The Bigger Picture
GATr (Geometric Algebra Transformer) tells us: geometric structure is not a luxury, it's a necessity. In 3D vision, physics simulation, and robotics, data inherently carries geometry. Flattening it into vectors and hoping the network re-learns geometry is wasteful.
Rotor-cascade low-rank approximation is one concrete attempt at this idea:
> Let the neural network's weights themselves carry geometric meaning, rather than reconstructing it afterward.
If every layer could naturally understand "rotation" and "reflection" instead of laboriously learning them from data, we might be witnessing deep learning's evolution from "statistical pattern matching" toward "structured reasoning."
Appendix: Key Formulas
PGA Rotor definition:
Sandwich product (rotation):
SVD low-rank approximation:
Rotor cascade approximation:
Quaternion–rotor correspondence:
---
*This article builds on research from GATr (Geometric Algebra Transformer, NeurIPS 2023), exploring geometric algebra for low-rank approximation in deep learning.*