Causal Grassmann Sequence Modeling Architecture: Replacing Self-Attention with Geometry

Forum topic · 小凯 · 2025-12-24

Summary

This post introduces a Causal Grassmann sequence modeling architecture that replaces the Transformer's self-attention mechanism with geometric operations on the Grassmann manifold Gr(m, D). Local token pairs are projected into a low-dimensional space via a learned linear map, and their relationships are encoded using Plücker coordinates—2x2 sub-determinants that remain invariant under basis changes of the underlying subspace. A gated fusion mechanism then mixes these geometric features back into the high-dimensional hidden states. The architecture achieves O(Lmr²) linear complexity in sequence length, compared to the O(L²d) quadratic cost of dense attention, while operating on a finite-dimensional manifold whose geometric invariants are easier to define and track, improving interpretability. The post includes a detailed comparison table with Transformers covering mechanism, complexity, explainability, performance, advantages, and challenges. Preliminary experiments with 13-18M parameter models report results competitive with Transformers, including 85.5% accuracy on natural language inference, with slight gains on some tasks. It also discusses application prospects and open challenges such as limited maturity and unverified large-scale performance.

Introduction

The Causal Grassmann sequence modeling architecture proposes an alternative to self-attention by modeling token relationships as geometric flows on the Grassmann manifold, using Plücker coordinates to capture and fuse local dependencies. Unlike attention's dense, high-dimensional weight matrix, this approach is structured and geometric, offering O(L) linear complexity and improved interpretability by moving core computations onto a finite-dimensional manifold with clear mathematical structure.

Key points

Grassmann manifold Gr(m, D): the space of all m-dimensional linear subspaces of D-dimensional Euclidean space, itself a smooth manifold. The architecture mainly uses Gr(2, r) (2D subspaces of r-dimensional space).
Dimension formula: dim(Gr(m, D)) = m(D - m); for Gr(2, r) this is 2(r−2), far smaller than the L×L attention matrix.
Plücker coordinates: homogeneous coordinates representing linear subspaces; for a 2D subspace in r-dimensional space, a C(r, 2)-dimensional vector computed from 2×2 sub-determinants of the matrix formed by local token pairs:
p_ij = det([z_{t-Δ, i}, z_{t, i}; z_{t-Δ, j}, z_{t, j}]), for 1 ≤ i < j ≤ r.
They encode oriented areas (projected geometry of token pairs) and are invariant under basis changes of the subspace, so the model tracks geometric entities rather than specific vector representations.

Causal Grassmann mixing layer

1. Dimensionality reduction: project the hidden state to a low-dimensional space via a learnable map:

z_t = W_down × h_t ∈ R^r

Causal local pair formation

{(z_{t-Δ}, z_t) | Δ ∈ W}

Geometric feature fusion

h'_t = u_t × h_t + (1-u_t) × (W_p × p_t)

Flow: h_t ∈ R^d → W_down → z_t ∈ R^r → causal pairs → Plücker coordinates → geometric features p_t ∈ R^C(r,2) → gated fusion → h'_t ∈ R^d.

Comparison with Transformer self-attention

| Dimension | Transformer (self-attention) | Causal Grassmann (mixing layer) | |---|---|---| | Core mechanism | Attention weights over all token pairs, forming a dense L×L matrix | Local token pairs mapped to points on the Grassmann manifold, encoded via Plücker coordinates | | Complexity | O(L²d) — quadratic, limits long sequences | O(Lmr²) — linear in sequence length | | Interpretability | Low: unstructured high-dimensional attention tensors, no global invariants | High: finite-dimensional manifold enables defining and tracking global geometric invariants | | Performance (preliminary) | Excellent across tasks, especially at large scale | Competitive at small/medium scale (13-18M parameters), slightly ahead on some tasks | | Main strengths | Powerful global dependency modeling, mature ecosystem | Efficiency, interpretability, novel geometric perspective | | Main challenges | High compute/memory cost, poor interpretability | Low maturity, unverified large-scale performance, requires specialized engineering |

Experimental highlights

Model scale: 13-18M parameters
Natural language inference (NLI) accuracy: 85.5%
Performance comparable to Transformers at this scale, with slight gains on specific tasks.

Geometric invariance and interpretability

Because Plücker coordinates are invariant to basis transformations of the underlying subspace, the architecture exposes stable geometric invariants across layers and sequences—an interpretable handle on global model behavior that dense attention tensors lack. This provides a concrete path toward analyzing what the model tracks geometrically as it processes a sequence.

Outlook and challenges

Prospects: efficient long-sequence modeling, interpretable architectures, and a new geometric lens on sequence modeling.
Challenges: architectural maturity, scaling behavior beyond tens of millions of parameters remains unproven, and practical deployment requires dedicated engineering implementations.