When Geometric Algebra Meets MoE Routing: A Mathematical Experiment on Direction
Forum topic · 小凯 · 2026-04-14
Summary
This exploratory essay investigates replacing conventional softmax routing in Mixture-of-Experts (MoE) models with Geometric Algebra (GA)-based routing. Traditional routers compute similarity via dot products between token vectors and expert vectors, treating routing as a nearest-neighbor classification. The author proposes representing tokens and experts as multivectors in Projective Geometric Algebra (PGA), inspired by the GATr architecture, and using geometric inner products or rotors instead of inner-product similarity. Three potential advantages are analyzed: geometric consistency via grade matching, continuous routing via rotor interpolation, and E(3) equivariance for rotation-symmetric data such as 3D point clouds and molecules. Four obstacles are honestly assessed, including 16× computational overhead from multivector representations, the discreteness gap between continuous geometric scores and Top-K sparse selection, training instability from grade explosion in geometric products, and the unclear geometric identity of learned experts. A staged research roadmap is proposed, beginning with a proof-of-concept on simple 2D transformations, followed by a hybrid approach that keeps a standard Transformer backbone and only replaces the router, and finally a fully geometric MoE stack.
Key points
- Problem framing: Conventional MoE routing computes
scores = softmax(W @ x) and selects Top-K experts based on vector inner-product similarity, treating expert selection as a nearest-neighbor problem that ignores geometric structure.
- Geometric Algebra (GA) background: Inspired by GATr, tokens and experts are represented as 16-dimensional multivectors in PGA. Geometric objects such as points, lines, planes, and rotors (rotations) become algebraic entities manipulated via the geometric product and the sandwich product
rotor ⊗ point ⊗ rotor_inverse, replacing rotation matrices and trig functions.
- Proposed GA routing sketch: Replace softmax with a geometric inner product
<token_mv, expert_mv>_GA between multivectors. Experts could be parameterized as bivectors (rotation planes) or trivectors (positions/planes), and token embeddings are projected into PGA space. Rotor interpolation (slerp) offers a continuous alternative to discrete Top-K selection.
- Three claimed advantages:
1. *Geometric consistency* via grade matching, preventing nonsensical matches such as rotation-vs-translation pairings that purely directional inner products may produce.
2. *Continuous routing* through rotor composition, enabling soft expert blending rather than hard binary selection.
3. *Rotation equivariance/invariance* inherited from E(3)-equivariant operations, useful for 3D point clouds, molecular structures, and robotics.
- Four honest obstacles:
1. *Compute overhead*: 16× parameter and activation cost compared with scalar transformers; geometric products are more expensive than matmul.
2. *Discreteness gap*: Top-K sparsity is hard to recover from continuous geometric scores; possible fixes include Gumbel-Softmax relaxation or abandoning sparsity for a Dense-MoE variant.
3. *Training instability*: geometric products mix all grades and can explode numerically; careful LayerNorm-style normalization is required, and routers may be harder to stabilize than softmax.
4. *Expert geometric identity*: unclear how experts should acquire or be assigned geometric meaning—random initialization, predefined prototypes, or meta-learning are all unexplored.
- Proposed research roadmap:
1. *Proof of concept*: 8-expert / Top-2 toy MoE learning 2D rotations and translations, comparing softmax routing vs GA inner-product routing on rotation-symmetric data, with routing-weight visualization and stability analysis.
2. *Hybrid approach*: keep a standard Transformer backbone; only replace the router with multivector embeddings and geometric routing, then re-enter the normal Top-K pipeline.
3. *Full geometric MoE*: geometrically constrained expert parameters, geometric versions of auxiliary load-balancing loss, and multivector-aware all-to-all communication for distributed training.
- Reflections and caveats: The author admits uncertainty about whether semantic (language) space actually has useful geometric symmetry, whether training will be stable, and whether expert multivectors can be learned meaningfully. The piece is framed as a thought experiment aimed at escaping the Softmax+Top-K comfort zone and asking whether routing should be a geometric transformation problem rather than a classification problem.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618462