Summary
This paper presents an exact, component-by-component quantum realization of softmax attention for problems constrained to the probability simplex, where inputs and outputs sum to one. The authors map each operation of the attention mechanism onto quantum primitives. Attention scores become Hadamard-test statistics on block-encoded projections of amplitude-encoded inputs. The exponential softmax is represented as the interior of a cosine-squared family produced by Born-rule measurements under an exact bijection, with its boundary encoding sparse attention with exact zeros at finite parameter values. Softmax temperature is realized as a repetition count using post-selected measurement rounds that produce discretized inverse temperature exactly. Value aggregation is a deterministic column-loading channel extending column-stochastic value matrices, and gated residuals correspond to a single ancilla qubit preparation angle with an additive identity at the mixing angle π/2. Every learnable parameter becomes a rotation-gate angle. Composite layers are exact in the infinite-sample limit, with each attention score mapped to a measure-and-reload step. A fully coherent variant uses quantum singular value transformation to achieve ε-approximation in the infinite-depth limit. The algebraic core is machine-verified in Lean 4.
Quantum Roadmap for Softmax Attention
This work establishes an exact quantum analog of softmax attention for the subclass of attention problems where inputs and outputs are constrained to the probability simplex (summing to one).
Key Points
- Attention scores: Hadamard-test statistics computed on block-encoded projections of amplitude-encoded inputs.
- Exponential softmax: Interior of a cosine-squared family generated by Born-rule measurement under an exact bijection. The boundary of this family expresses sparse attention with exact zeros at finite parameter values.
- Softmax temperature: A repetition count where post-selected measurement rounds realize a discretized inverse temperature exactly.
- Value aggregation: A deterministic column-loading channel that extends column-stochastic value matrices.
- Gated residuals: A single ancilla qubit preparation angle, with an additive identity at the mixing angle π/2.
- Learnable parameters: Each maps directly to a rotation-gate angle.
- Composite layers: Exact in the infinite-sample limit, with each attention score corresponding to a measure-and-reload step.
- Fully coherent variant: Uses quantum singular value transformation (QSVT) to achieve ε-approximation in the infinite-depth limit.
- Verification: The algebraic core is machine-verified in Lean 4.
Source
- arXiv: 2608.11173
- Authors: Eric A. F. Reinhardt, Adam J. Hauser
- Field: Machine Learning
- Posted: 2026-08-11
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633407