Versor Deep Dive: Geometric Product Attention and Recursive Rotor Accumulator for Sequence Modeling

Forum topic · 小凯 · 2026-04-01

Summary

A detailed analysis of Versor, a geometric sequence architecture based on conformal geometric algebra Cl(4,1), introduced in the paper by Edward Hirst and Truong Minh Huy. The architecture introduces two core innovations: Geometric Product Attention (GPA), which replaces dot-product attention with the full geometric product—preserving both scalar (distance/similarity) and bivector (direction/torque) information with a learnable balance parameter gamma—and the Recursive Rotor Accumulator (RRA), which models sequences as continuous rotation trajectories, achieving O(L) linear complexity instead of O(L^2). Versor reports strong empirical results: 99.3% MCC versus 50.4% for ViT on a zero-shot scale-generalization task, 1/200 the parameters of a Transformer with better accuracy on chaotic N-body dynamics, and 10x heavier particle masses handled with improved error (-19.9%) where Transformers fail catastrophically (+3097.2%). Hardware-level bit-masked kernels and matrix isomorphism kernels deliver a 78x speedup for geometric product computation. The article includes PyTorch implementations and comparisons with GATr.

Overview

Versor is a geometric sequence architecture that replaces the Transformer's dot-product attention with geometric product attention over conformal geometric algebra Cl(4,1), and replaces quadratic self-attention with a Recursive Rotor Accumulator achieving O(L) complexity. This post is a deep-dive analysis of the paper *Versor: A Geometric Sequence Architecture* (arXiv:2602.10195) by Edward Hirst and Truong Minh Huy.

Key points

From dot product to geometric product. Dot-product attention yields only a scalar similarity score, discarding directional information. Versor's Geometric Product Attention (GPA) decomposes the geometric product of Query and (reversed) Key into a scalar part (grade-0, similarity/distance) and a bivector part (grade-2, directional coupling), combined via a learnable parameter gamma:

\alpha_{ij} = softmax\left(\frac{\langle Q_i \widetilde{K}_j \rangle_0 + \gamma \|\langle Q_i \widetilde{K}_j \rangle_2\|}{\sqrt{d_{in}}}\right)

The analogy: the scalar part captures "Coulomb-like" distance effects, the bivector part captures "Lorentz-like" directional effects.

Recursive Rotor Accumulator (RRA). Instead of treating tokens independently, RRA models a sequence as a continuous rotation trajectory, updating a cumulative rotor:

R_{total} \leftarrow R_{total} \cdot \Delta R_i

Each new token requires only one geometric product with the accumulated rotor—O(1) per token, O(L) overall versus O(L²) for Transformers. Rotors encode relative geometric relations (SE(3)-equivariant), not absolute positions, so information is not lost.

Conformal Geometric Algebra Cl(4,1). The 5-dimensional CGA (3 Euclidean + origin + infinity basis vectors) natively supports rotations, translations, scaling, and reflections, with SE(3)-equivariance built into the algebra. This enables zero-shot scale generalization: on a "broken snake" task, Versor reaches 99.3% MCC at unseen grid resolutions versus 50.4% for Vision Transformers, because it learns resolution-independent algebraic rules (e.g., \|Δx\| > 1 signals a gap) rather than pixel coordinates.
Hardware optimization. Direct geometric product computation over 32 basis elements (2^5) is expensive (~5-10x the constant factor of matrix multiplication). Versor's bit-masked kernels encode basis elements as 5-bit masks, using XOR/AND bitwise operations and popcount-based sign computation, achieving 78× speedup (82 ms → 1.05 ms) and >50× memory reduction. Matrix isomorphism kernels further map geometric products onto optimized GPU matrix multiplication via the isomorphism between Cl(4,1) and 4×4 matrix algebra.

Experimental results

| Task | Versor | Transformer | GATr | |------|--------|-------------|------| | Chaotic N-body MSE (params) | 5.21 (0.007M) | 6.61 (1.320M) | 8.32 (~0.1M) | | Energy drift | 133.0% | 381.1% | 173.8% | | Zero-shot N=3/7 particles | Stable error | Complete failure | — | | Hidden velocity (MSE) | 0.003 | — | 0.3253 | | 10× out-of-distribution mass | −19.9% error | +3097.2% error | — |

Versor achieves better prediction accuracy with roughly 1/200 the parameters, one-third the energy drift, and robust zero-shot generalization across system size, scale, and mass distribution shifts.

Comparison with GATr

| Aspect | GATr | Versor | |--------|------|--------| | Algebra | Cl(3,0,1) PGA | Cl(4,1) CGA | | Attention | Geometrized dot product | Full geometric product | | Complexity | O(L²) | O(L) | | Scaling equivariance | No | Yes | | Position encoding | Required | Not required |

PGA supports rotations and translations but not scaling; CGA's conformal transformations preserve angles and include scaling—explaining Versor's advantage on scale-generalization tasks.

Implementation sketches

GPA projects inputs into 32-dimensional CGA multivectors per head, extracts grade-0 and grade-2 norm components of the geometric product, and combines them with learnable gamma. RRA maintains a cumulative rotor state updated per token with a geometric product, applying the sandwich product to each input. Bit-masked kernels implement basis-blade combination via a_bits ^ b_bits (outer) and a_bits & b_bits (inner), with signs from popcount parity.

Significance

Versor challenges three Transformer assumptions—data as vectors, attention as dot product, position as something to encode—replacing them with multivectors, geometric products, and implicit relative encoding. Reported applications span physical simulation, robotics (SE(3)-equivariant motion planning), and 3D vision, with future directions including specialized geometric processing units and LLM-scale validation.

*Based on the paper "Versor: A Geometric Sequence Architecture" (arXiv:2602.10195) by Edward Hirst and Truong Minh Huy.*