Versor Deep Dive: Geometric Product Attention and Recursive Rotor Accumulator in Conformal Geometric Algebra

Overview

Versor is a geometric sequence architecture that replaces the Transformer's dot-product attention with Geometric Product Attention (GPA) and its quadratic self-attention with a Recursive Rotor Accumulator (RRA). It operates entirely within Conformal Geometric Algebra (CGA), Cl(4,1), a 32-basis-element algebra over a 5-dimensional vector space (3 Euclidean basis vectors, an origin e0, and a point at infinity e∞).

The article below is a technical deep dive based on the paper *Versor: A Geometric Sequence Architecture* (arXiv:2602.10195) by Edward Hirst and Truong Minh Huy.

Key points

1. Geometric Product Attention (GPA)

• Traditional attention uses only the dot product, producing a scalar similarity score that discards directional information.
• GPA uses the full geometric product of Query and reversed Key multivectors, decomposed into:
• Grade-0 (scalar) part ⟨QK̃⟩0 — similarity/distance
• Grade-2 (bivector) part ⟨QK̃⟩2 — directional coupling (analogous to torque)
• Attention formula:
\[\alpha_{ij} = softmax\left(\frac{\langle Q_i \widetilde{K}_j \rangle_0 + \gamma \|\langle Q_i \widetilde{K}_j \rangle_2\|}{\sqrt{d_{in}}}\right)\]

where γ is a learnable parameter balancing distance and direction. This yields interpretable attention weights with clear geometric meaning.

2. Recursive Rotor Accumulator (RRA)

• Instead of comparing every token with all history (O(L²)), RRA maintains a single accumulated rotor updated by one geometric product per token:
\[R_{total} \leftarrow R_{total} \cdot \Delta R_i\]
• Each update is O(1), giving O(L) linear complexity overall.
• Rotors encode *relative* geometric transformations rather than absolute positions, so the representation is SE(3)-equivariant by construction.

3. Conformal Geometric Algebra Cl(4,1)

• CGA natively supports rotations, translations, scaling, and reflections (conformal transformations), which PGA (Cl(3,0,1), used by GATr) cannot represent scaling.
• Conformal embedding of a point: P = x + ½x²e∞ + e0
• Rotor for rotation: R = cos(θ/2) − sin(θ/2) B̂
• Because Versor learns algebraic rules on displacement vectors (e.g., the condition ‖Δx‖ > 1 is resolution-independent), it achieves zero-shot scale generalization without positional encodings.

4. Hardware optimization

• Direct geometric products over 32 basis elements have a 5–10× constant-factor penalty vs. matrix multiplication.
• Bit-masked kernels encode basis elements as 5-bit masks, using bitwise XOR/AND and popcount-based sign computation, exploiting sparsity.
• Matrix isomorphism kernels map the geometric product to optimized matrix multiplications usable with GPU tensor cores.
• Reported: latency reduced from 82 ms to 1.05 ms (78× speedup), memory reduced by >50×.

5. Experimental results

| Task | Versor | Baseline | |------|--------|----------| | Broken-snake zero-shot scale generalization | 99.3% MCC | ViT: 50.4% | | Chaotic N-body MSE (0.007M params) | 5.21, 133.0% energy drift | Transformer (1.32M params): 6.61 MSE, 381.1% drift; GATr: 8.32, 173.8% | | Hidden-velocity inference | 0.003 MSE | GATr: 0.3253 MSE | | Out-of-distribution 10× mass | error improves −19.9% | Transformer: +3097.2% error | | Variable system size (N=3–7, trained on N=5) | stable zero-shot generalization | Transformer fails due to fixed input dimensions |

6. Comparison with GATr and Transformers

| Feature | Transformer | GATr | Versor | |---------|-------------|------|--------| | Algebra | ℝᵈ | Cl(3,0,1) PGA | Cl(4,1) CGA | | Attention | dot product | geometricized dot product | full geometric product | | Complexity | O(L²) | O(L²) | O(L) | | Positional encoding | required | required | none | | Scale equivariance | no | no | yes |

7. Implementation sketches (from the article)

The original post includes PyTorch-style reference implementations:

• GeometricProductAttention: projects inputs to 32-dim CGA multivectors per head, extracts grade-0 and grade-2 norm components, and combines them with the learnable γ.
• RecursiveRotorAccumulator: maintains an accumulated rotor buffer, normalizes each input to a delta rotor, and applies the sandwich product.
• BitMaskedCGAKernel: computes geometric products via 5-bit masks, XOR/OR bit operations, and popcount-based sign determination.

8. Significance and outlook

• Versor challenges Transformer assumptions: data as multivectors, attention as geometric product, positions as relative transformations.
• It marks geometric deep learning moving from engineering heuristics toward a unified mathematical framework.
• Suggested applications: physical simulation (molecular dynamics, climate), robotics (SE(3)-equivariant motion planning), and 3D vision.
• Future directions: dedicated geometric processing units (GAPUs), Riemannian optimization, relativistic/quantum extensions, and validation at LLM scale.

Conclusion

Versor demonstrates that replacing dot products with the full geometric product, and quadratic attention with recursive rotor composition, yields a model that learns algebraic structure rather than pixel coordinates — achieving dramatic parameter efficiency, linear-time sequence modeling, and zero-shot generalization across scales and system sizes.

*Based on the paper "Versor: A Geometric Sequence Architecture" (arXiv:2602.10195) by Edward Hirst and Truong Minh Huy.*