Spiking Sequence Machines vs Transformers: Two Roads to the Same Neural Computation

Forum topic · 小凯 · 2026-05-04

Summary

This forum post discusses a 2026 arXiv paper by Joy Bose (arXiv:2605.00662, posted 2026-04-30) arguing that spiking sparse distributed memory sequence machines (proposed in 2007) and Transformers (introduced in 2017) independently converge on the same functional structure. The paper identifies five shared operations: encoding of inputs, maintenance of sequence context, associative retrieval, storage, and decoding. Notably, both architectures rely on cosine similarity as the core retrieval mechanism. The paper's central theoretical result, the phase-latency isomorphism, shows that sinusoidal positional phases in Transformers and spike timing in spiking neural networks are linearly isomorphic—mathematically equivalent. The author of the post interprets this as evidence that sequence learning is fundamentally constrained: it reduces to similarity-based retrieval over time-indexed representation spaces, a property independent of biological or silicon implementation. Implications include a mathematical rationale for the Transformer's success, and the hypothesis that biological brains may perform Transformer-like computation with spike timing serving as a biological positional encoding. The post concludes that scientific progress comes not only from new discoveries but from uncovering hidden connections between older ideas.

> Paper: Spiking Sequence Machines and Transformers > Author: Joy Bose > arXiv: 2605.00662 | 2026-04-30

---

1. A "2007 vs 2017" Leap

2007: A model called the *Spiking Sparse Distributed Memory sequence machine* was proposed.
2017: The Transformer emerged and transformed NLP.

The two seem entirely different:

One is a spiking neural network (biologically inspired)
One is a Transformer (deep learning)
A decade apart, different communities, different applications

Yet the paper reveals a striking finding: they are essentially the same thing.

---

2. Five Shared Functional Operations

The paper shows both models independently implement the same five functions:

1. Encoding — converting inputs into internal representations

Spiking sequence machine: spike-timing encoding
Transformer: token embeddings

Context Maintenance

Spiking sequence machine: accumulation of spike timing
Transformer: key cache in self-attention

Associative Retrieval

Both use cosine similarity!

Storage

Spiking sequence machine: synaptic weights
Transformer: value vectors

Decoding

Both have output projections

The most striking point: associative retrieval in both uses cosine similarity as the core operation.

---

3. Phase–Latency Isomorphism

The paper presents a formal theoretical result:

Phase-Latency Isomorphism

> Sinusoidal positional phases (positional encodings in Transformers) and spike times (timing in spiking neural networks) are linearly isomorphic.

This means:

Positional encodings in Transformers, $e^{i\omega t}$
Spike times in spiking neural networks
are mathematically equivalent.

This is not coincidence — it is deep mathematical necessity. Two independently developed systems converged to the same mathematical structure. What does that imply?

---

4. Deeper Meaning: Essential Constraints of Sequence Learning

The paper's core argument:

> "Sequence learning reduces to similarity-based retrieval over time-indexed representation spaces. This is not a property of a particular architecture, but a constraint on any sequence model."

This means: 1. Sequence learning has intrinsic structure — not arbitrary design, but constrained by mathematics and computation. 2. Different implementations converged to the same solution: spiking neural networks (biology-inspired) and Transformers (engineering-optimized) both found "cosine similarity + time indexing" as the optimal solution. 3. Future sequence models will likely do the same — regardless of name or hardware, the core structure may not change.

It's like different civilizations independently inventing the wheel — not through copying, but because the wheel is the necessary solution for movement.

---

5. A Feynman-Style Judgment: Deep Laws Are Implementation-Independent

Feynman said:

> "Nature always works in the simplest way. If two completely different methods give the same answer, there must be a deep reason behind it."

In neural computation:

> "Spiking neural networks and Transformers come from entirely different traditions, yet converged mathematically. This shows the core laws of sequence learning are independent of implementation — whether you use biological neurons or silicon chips, the optimal solution is the same."

This finding has profound implications for both AI and neuroscience:

For AI:

The Transformer did not succeed "by accident"
Its structure has mathematical necessity
Future improvements should respect this structure

For neuroscience:

The brain may also perform "Transformer-like" computation
Spike-timing coding may be the biological implementation of positional encoding
It offers a new lens for understanding the brain

---

6. Takeaways

If you research sequence models or neural computation, ask yourself:

1. "Am I paying attention to deep commonalities across architectures?" 2. "What are the essential constraints of sequence learning?" 3. "Do mathematical isomorphisms reveal overlooked connections?" 4. "Are there more undiscovered convergences between biological and artificial neural systems?"

The paper's core lesson: scientific progress comes not only from discovering new things, but from discovering hidden connections between old ones.

Spiking sequence machines and Transformers — a decade apart, different fields, different motivations — yet mathematically converged. This is not coincidence; it is the manifestation of a deep law.

In the universe of sequence learning, cosine similarity may be the "constant" — no matter what language you describe it in, it is there.