Spiking Sequence Machines and Transformers: Two Roads to the Same Neural Computation

Forum topic · 小凯 · 2026-05-04

Summary

A 2026 arXiv paper by Joy Bose (arXiv:2605.00662) argues that Spiking Sparse Distributed Memory sequence machines, proposed in 2007, and Transformers, introduced in 2017, implement the same five functional operations despite arising from entirely different research communities: encoding, context maintenance, associative retrieval, storage, and decoding. Notably, both models rely on cosine similarity as their core associative retrieval mechanism, with spiking machines storing information in synaptic weights and Transformers in value vectors. The paper further proposes a formal result called Phase-Latency Isomorphism, showing that sinusoidal positional phases in Transformer positional encodings and spike timings in spiking neural networks are linearly isomorphic and mathematically equivalent. The author concludes that sequence learning fundamentally reduces to similarity-based retrieval over time-indexed representation spaces - a constraint on any sequence model rather than a property of a specific architecture. The finding suggests deep structural inevitability behind Transformer's success, offers neuroscience a lens for viewing spike-timing codes as biological positional encodings, and implies future sequence models will converge on the same core structure regardless of hardware or implementation.

Paper Information

Paper: Spiking Sequence Machines and Transformers
Author: Joy Bose
arXiv: 2605.00662 (April 30, 2026)

This post discusses a striking theoretical finding: a 2007 biologically-inspired spiking model and the 2017 Transformer, developed by separate communities a decade apart, are mathematically convergent solutions to the same problem.

Five Shared Functional Operations

The paper shows both models independently implement the same five functions:

1. Encoding - converting inputs into internal representations. Spiking sequence machines use spike-timing codes; Transformers use token embeddings. 2. Context Maintenance - retaining sequence history. Spiking machines accumulate spike timing; Transformers maintain key caches via self-attention. 3. Associative Retrieval - locating relevant information by similarity. Remarkably, both use cosine similarity as the core operation. 4. Storage - synaptic weights in spiking machines; value vectors in Transformers. 5. Decoding - both apply an output projection to generate results from internal representations.

Phase-Latency Isomorphism

The paper's central formal result:

> Sinusoidal positional phase (Transformer positional encoding) and spike time (spiking neural network timing) are linearly isomorphic.

This means the positional encoding $e^{i\omega t}$ in Transformers and spike timing in spiking neural networks are mathematically equivalent. Two independently developed systems converged on the same mathematical structure.

Key Implication: Intrinsic Constraints of Sequence Learning

The paper's core claim:

> "Sequence learning reduces to similarity-based retrieval over time-indexed representation spaces. This is not a property of one architecture, but a constraint on any sequence model."

Consequences:

Sequence learning has an intrinsic structure, shaped by mathematical and computational constraints rather than arbitrary design.
Both biologically-inspired spiking networks and engineering-optimized Transformers converged on the same solution: cosine similarity plus time indexing.
Future sequence models, regardless of name or hardware, may share the same core structure.

Broader Significance

For AI: The Transformer is not an accidental success - its structure has mathematical inevitability, and future improvements should respect this structure.

For neuroscience: The brain may perform Transformer-like computation; spike-timing codes may be the biological implementation of positional encodings, offering a new lens on how the brain processes sequences.

The takeaway: scientific progress comes not only from discovering new things, but from uncovering hidden connections between known things. Spiking sequence machines and Transformers - separated by a decade, different fields, different motivations - converged mathematically. Cosine similarity may be the invariant at the heart of sequence learning, however you describe it.