A Strange Intuition
Imagine holding a Rubik's cube.
The traditional approach describes its state as a list: the colors of all 54 stickers—red here, blue there, yellow on the back. The alternative is a *transform-based* description: record the rotations it underwent. Starting from solved, rotate 30° about the x-axis, 45° about a diagonal, then the top layer 90° clockwise. One describes results; the other describes process.
Low-rank approximation in deep learning has long been stuck in the first mindset.
SVD: A Great List
SVD (Singular Value Decomposition) is a mathematical marvel. Any matrix \(A\) can be factored as:
The diagonal entries of \(\Sigma\)—the singular values—tell you in which directions the transformation stretches most.
But singular values have a problem: they are scalars.
It's like describing an elastic fabric by flattening it, recording how much each cell stretched, and sewing it back. The geometry—the bends, twists, rotations—gets reduced to pure numbers. That works in many settings. But what if your data has intrinsic geometric meaning? Think 3D point clouds, molecular conformations, rigid-body motion.
Geometric Algebra: Recovering Lost Structure
In the 19th century, William Clifford invented an algebra that treats vectors, rotations, and scaling in a unified way. In Geometric Algebra, a *multivector* can simultaneously contain:
- Scalars: pure numbers
- Vectors: directed quantities
- Bivectors: oriented rotation planes (a flagpole is the axis, the flag's area is the angle)
- Trivectors: volume elements
- Better continual learning
- Smoother latent space structure
- Potentially better generalization
In 3D space, these combine into exactly a 16-dimensional space. Strange—but here rotation is native.
Rotor: The DNA of Rotation
In geometric algebra, rotations are represented by a Rotor:
where \(B\) is a bivector encoding the rotation plane and angle. Crucially, rotors compose: one rotor is one rotation; their product is rotation after rotation. A chain of rotors is like a strand of DNA encoding a complex motion.
Core idea:
> Instead of approximating a linear transform with scalar singular values, encode it with a cascade of rotors.
The Sandwich Product
A rotor acts on a vector via the sandwich product:
No matter how complex \(R\) is, if it is a unit rotor, this operation is guaranteed to be an orthogonal transformation—lengths and angles preserved, a pure geometric rotation.
Compare with SVD: linear interpolation between two SVD decompositions is generally not a valid transform—singular values can go negative and geometry breaks down. But interpolating between rotors (SLERP) always yields a valid rotation.
Experiments: Theory vs. Reality
Experiment 1: Rotation Transform
Learning a 3D rotation:
| Method | Params | Test Error | Train Time | |--------|--------|-----------|------------| | Standard MLP | 115 | 2.52 | 0.028s | | SVD low-rank (rank=2) | 91 | 2.40 | 0.031s | | Rotor cascade (2 rotors) | 411 | 2.52 | 0.125s |
A sobering result: the rotor approach is parameter-inefficient. In the current design, rotors live in a 16-dimensional PGA (Projective Geometric Algebra) space, and projecting inputs into it costs \(16 \times d_{in}\) parameters—huge overhead at small dimensions.
Experiment 2: Function Fitting
Fitting a random linear function: rotor methods reach comparable accuracy but need more parameters and time.
Experiment 3: Parameter Efficiency
| Config | SVD Params | Rotor Params | Ratio | |--------|-----------|--------------|-------| | (10, 32, 5, rank=2) | 86 | 760 | 8.8× | | (10, 32, 5, rank=8) | 344 | 784 | 2.3× | | (100, 128, 50, rank=8) | 1832 | 3952 | 2.2× |
Interestingly, the relative overhead shrinks as dimensions grow: the projection cost is fixed while SVD parameters grow linearly.
Why Not Give Up?
Three reasons the direction is worth pursuing:
1. Geometric Consistency
Singular values carry no geometric meaning. A rotor corresponds to a real rotation. Saying "this transform is approximated by 3 rotors" means "it decomposes into 3 simple geometric motions"—valuable wherever interpretable transforms matter (robot motion planning, physics simulation).
2. Interpolation and Generalization
Straight-line interpolation in parameter space between trained SVD networks is usually meaningless. Rotor interpolation stays geometrically valid, suggesting:
3. Equivariance
Rotor decomposition naturally preserves E(3) equivariance (rotation, translation, reflection). For tasks demanding this symmetry—molecular property prediction, physical simulation—rotors may be more elegant than forcing constraints onto SVD.
Next Step: The Return of Quaternions
The bottleneck is clear: the 16-dimensional PGA projection overhead. But for pure rotations, a rotor is equivalent to a quaternion:
Quaternions need only 4 parameters instead of 16. Next step: represent rotors directly as quaternions, eliminating the PGA projection and closing the parameter-efficiency gap with SVD while keeping the geometric advantages.
The Bigger Picture
GATr teaches us: geometric structure is not a luxury, but a necessity. In 3D vision, physics simulation, and robotics, data carries geometry intrinsically. Flattening it into vectors and hoping the network relearns geometry is wasteful.
> Let the network's weights themselves carry geometric meaning, rather than reconstructing it after the fact.
What if every layer naturally understood "rotation" and "reflection" instead of laboriously learning them from data? Perhaps we are witnessing deep learning evolve from statistical pattern matching toward structured reasoning.
Appendix: Key Formulas
PGA Rotor:
Sandwich product:
SVD low-rank approximation:
Rotor cascade approximation:
Quaternion–rotor correspondence:
---
*This post builds on GATr (Geometric Algebra Transformer, NeurIPS 2023), exploring geometric algebra for low-rank approximation in deep learning.*