[论文] Reading Why: Born-Measured Attention with Exact Word Evidence on Light...

论文概要

研究领域: NLP
作者: Teerawat Tongloy, Jiraporn Pongsopa, Rangsan Jomtarak
发布时间: 2026-11-03
arXiv: 2611.05355

中文摘要

当语言模型作答时，我们通常只能估计「为什么」。我们构造了一个每一步都是 Born 测量的 Transformer：注意力为 |⟨q|k⟩|⁴ 按行归一化，前馈非线性被替换为「被测乘积」，读出也是 Born 型。由于分母在输入处取值，每个类别概率是词幅值的多项式，因此其积分梯度证据可由有限次求积精确算出、无求积误差：词对类别概率超出无信息份额的贡献之和可精确到 10⁻¹¹ 以上。在 e-SNLI 自然语言推理上，该模型在嵌入之外约 98k 参数下取得 76.98±0.17% 的测试准确率，与同规模 softmax Transformer（76.62±0.36%）持平。证据排名第一的词被删除时在 45% 的测试对上会改变答案（任何单次删除能改变的上限为 53%），高于用同法解释的 softmax 模型（32%）及其 attention×gradient（40%）。注意力看起来更贴近标注者高亮的词，但对答案的解释力反而更弱。同一组权重可精确编译到光电子学器件与量子比特上：一层相干注意力在 23 个量子比特上仅需 852 个 CNOT 门即可等效整个模型。在 IBM ibm_kingston 上，仅通过 Hadamard 测试读出注意力重叠，设备在两个短测试对上给出了模型的答案，并在删除词时按模型方式保留或改变答案；证据本身尚不能在真机上存活，我们分析了在硬件上测它所需的采样数与标定。

原文摘要

When a language model answers, we can usually only estimate why. We build a transformer in which every step is a Born measurement - attention \(|\langle q|k\rangle|^4\) normalised over each row, a measured product in place of the feed-forward nonlinearity, and a Born readout. With its denominators held at their values for the input, each class probability is a polynomial in the word amplitudes, so its integrated-gradient evidence is computed by a finite quadrature with no quadrature error: the words' contributions sum to the class probability above its no-information share, to \(10^{-11}\) or better. On e-SNLI natural language inference, with about 98k parameters outside the embedding, the model reaches \(76.98 \pm 0.17\)% test accuracy, level with a softmax transformer of the same size (\(76.62 \pm 0.36\)%). Deleting the word its evidence ranks first changes the answer in 45% of test pairs, out of the 53% any single deletion can change, against 32% for the softmax model explained the same way and 40% by its attention x gradient. Attention can look more like the words annotators highlighted, yet accounts for the answer less well: deleting the word it ranks first changes the Born Transformer's answer in 32% of pairs. The same weights compile exactly to optoelectronic optics and to qubits, and a coherent attention layer becomes one circuit equal to the model (852 CNOTs on 23 qubits). On IBM's ibm_kingston, reading only the attention overlaps by Hadamard tests, the device gave the model's answers on two short test pairs and kept or changed them under deletion as the model does; the evidence itself did not yet survive the device, and we work out the shots and calibration that measuring it on hardware would need.

自动采集于 2026-11-05

#论文 #arXiv #AI #小凯