Transformer's Quadratic Complexity and Black-Box Problem: Alternative Approaches and the Causal Grassmann Transformer

Forum topic · ✨步子哥 · 2025-12-31

Summary

This forum post analyzes two core limitations of the Transformer architecture introduced in 'Attention Is All You Need' (Vaswani et al., 2017): the quadratic O(n²·d) computational complexity of self-attention, caused by computing the n×n attention score matrix, and the 'black-box' interpretability problem stemming from high-dimensional tensor lifting into L×L pairwise compatibility spaces. It surveys existing mitigation strategies: sparse attention (Longformer, BigBird, Reformer), linear attention and kernel approximations (Linformer, Performer, FlashAttention), white-box architectures like CRATE, and geometry/physics-grounded model design. The post then examines the paper 'Attention Is Not What You Need' (arXiv:2512.19428), which proposes the Causal Grassmann Transformer—a model that removes explicit attention entirely, reducing complexity from O(n²) to O(n) while offering intrinsic geometric interpretability via Grassmann manifolds. The model represents token states as evolving subspaces on a structured manifold rather than opaque attention tensors, addressing both efficiency and explainability challenges in a unified framework.

Introduction

Since the publication of *Attention Is All You Need* (Vaswani et al., 2017), Transformer-based models have become a cornerstone of NLP and deep learning at large. However, two inherent challenges have become increasingly prominent as models scale:

1. Quadratic computational complexity of self-attention 2. The "black-box" interpretability problem

The paper *Attention Is Not What You Need* (arXiv:2512.19428) proposes the Causal Grassmann Transformer as a unified solution to both challenges.

Key points

Self-attention computes relationships between all token pairs, generating an n×n attention score matrix: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V.
Complexity breakdown: QKᵀ matrix multiplication is O(n²·d_k), softmax is O(n²), and weighted summation is O(n²·d_v) — approximately O(n²·d) overall in time and space.
The "black-box" problem arises because the core operation is a "high-dimensional tensor lifting" of each token's d-dimensional hidden state into an L×L pairwise compatibility tensor space — mathematically intractable, with large degrees of freedom and no clear invariants describing global effects.

Existing alternatives

For quadratic complexity

Sparse attention: restrict each token's attention to a subset (Longformer: sliding window + global; BigBird: local + global + random; Reformer: locality-sensitive hashing) — near-linear complexity.
Linear attention: kernel-based softmax approximation exploiting associativity of matrix multiplication (Linformer: low-rank projection; Performer: FAVOR+; FlashAttention: IO-aware exact optimization).

For the black-box problem

White-box architectures: models like CRATE (Coding and Rate Reduction Transformer), where every layer has an explicit mathematical objective — maximizing coding rate reduction.
Geometry/physics-grounded design: replacing intractable high-dimensional tensor operations with evolution on manifolds with clear mathematical structure.

The Causal Grassmann Transformer

The paper's fundamental question: is the explicit L×L self-attention weight tensor truly necessary for powerful sequence modeling and reasoning?

Linear complexity: reduces O(n²) to O(n), significantly improving long-sequence processing efficiency.
Geometric interpretability: an intrinsically interpretable architecture based on Grassmann manifolds, where token states evolve as structured subspaces rather than opaque attention tensors.

Conclusion

The Causal Grassmann Transformer offers a creative combined answer to the two central limitations of the standard Transformer — quadratic complexity and lack of interpretability — by eliminating explicit attention in favor of geometrically structured state evolution.