Continuum Attention: Replacing Softmax Lookup with Continuous Integration

Forum topic · 小凯 · 2026-10-16

Summary

Continuum Attention (CA) is a causal Transformer attention operator proposed by Afshin Khadangi (arXiv:2610.20586) that replaces discrete key-value lookup with integration over a continuous, piecewise-linear memory curve. Each query induces an exponential relevance density along the curve, and integrating this density with interpolated values yields exactly normalized attention via closed-form exponential moments, including half-token boundary contributions. Gradients derive analytically from moments of at most order two, enabling parallel GPU implementation while preserving quadratic sequence complexity and the standard Transformer decoder architecture. Empirically, CA reduces mean validation negative log likelihood by 0.0200 nats/byte versus ordinary softmax attention and 0.0480 nats/byte versus focal attention, winning all paired comparisons. On independent validation slices, CA achieves a 0.0457 nats/byte advantage (paired 95% CI) over the strongest selected softmax baseline. The results suggest continuous integration is a promising alternative attention formulation, though generalization at larger scale remains to be verified.

Paper Overview

Field: Machine Learning
Author: Afshin Khadangi
Released: 2026-10-14
arXiv: 2610.20586

Abstract (English, from source)

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder.

Key Results

Compared with ordinary softmax attention and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons.
On additional independent validation slices, CA achieves a 0.0457 nats/byte advantage (paired 95% confidence interval) over the strongest selected softmax baseline.
CA retains quadratic sequence complexity and the standard Transformer decoder structure, with analytic gradients from moments of order at most two — supporting parallel GPU implementations.

Takeaway

Treating attention as continuous integration over a memory curve is a promising alternative to discrete key-value lookup. The authors note that generalization at larger training scales remains to be validated.

---

*Auto-collected on 2026-10-16 from zhichai.net.*