Caracal: Causal Architecture via Spectral Mixing — Attention-Free LLM with O(L log L) Long-Sequence Handling
> Paper: Caracal: Causal Architecture via Spectral Mixing > Authors: Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu > arXiv: 2605.00292 | 2026-04-29
1. The Architecture Bottleneck: "Attention Is Too Slow for Long Contexts"
Consider processing a long document:
The Transformer's dilemma:
- Attention: O(L²)
- L = 1,000 → 1 million operations
- L = 10,000 → 100 million operations
- A disaster for long texts
- Fixed length
- Poor extrapolation
- Weak long-sequence generalization
- O(L log L) or better complexity
- Natural sequence modeling
- No positional encodings
- Ability to handle ultra-long sequences
- Replaces attention
- Performs sequence mixing via FFT
- O(L log L) complexity
- Parameter-efficient
- Preserves autoregressive capability
- Asymmetric masking
- Causality enforced in the frequency domain, not the time domain
- FFT inherently handles sequence order
- No extra positional information required
- Natural length extrapolation
- Arbitrary lengths
- FFT-based sequence mixing
- Frequency-domain causal masking
- Efficient long-sequence modeling
- Transformer = everyone shakes hands with everyone else
- n people → n² handshakes → slow
- Caracal = people standing in a circle passing messages
- Information propagates like a "wave"
- Fast, with natural ordering
- Quadratic complexity O(L²): infeasible for long sequences, memory explosion, slow computation
- Positional encoding limits: fixed training length, poor long-sequence extrapolation, special tricks required
- Log-linear complexity O(L log L): long sequences become feasible, memory-friendly, fast
- Natural sequentiality: FFT is ordered, no positional encodings, arbitrary lengths, natural extrapolation
- Parameter efficiency: fewer parameters than attention, faster training, lightweight deployment
- Complex in the time domain → simple in the frequency domain
- Shift perspective → the problem dissolves
Limitations of positional encodings:
What's needed:
2. Caracal: A New Architecture Based on FFT Spectral Mixing
The paper proposes Caracal:
Core idea: > Replace attention with the Fast Fourier Transform (FFT) to achieve O(L log L) sequence mixing, while solving both the positional encoding and long-sequence problems.
Technical approach:
1. Multi-Head Fourier (MHF) module
2. Causal masking in the frequency domain
3. No positional encodings needed
4. Three contributions
An analogy:
3. Why FFT Beats Attention for Long Sequences
Attention's problems:
Caracal's advantages:
4. A Feynman-Style Takeaway: Sometimes the Answer Lies in Another Domain
Feynman noted that "knowing the name of something" and "understanding something" are entirely different.
In architecture design:
> "In the time domain, attention is O(L²) 'everyone looking at everyone.' Caracal's insight is that in the frequency domain, sequence mixing can be O(L log L) 'wave propagation' — sometimes changing the domain of the problem changes its complexity."
This reflects mathematical elegance:
5. Questions to Take Away
If you research LLM architectures or long-sequence modeling, ask yourself:
1. Is my model constrained by attention's quadratic complexity? 2. Are frequency-domain methods suited to sequence modeling? 3. Are positional encodings truly necessary? 4. Can FFT replace attention?
Caracal reminds us: architectural innovation lies not only in "improving attention" but in "moving beyond attention." As LLM architectures shift from time-domain attention to spectral mixing, they move from a quadratic bottleneck to log-linear freedom. The best future designs may not be attention variants, but a re-understanding of the nature of sequences: in the ocean of frequencies, wave propagation is more efficient than shaking hands one by one.