Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention — Paper Explained

Forum topic · 小凯 · 2026-08-20

Summary

This post explains the paper "Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention" (arXiv:2608.19171) by Sotirios P. Chatzis and Loukas Papadoulas. The paper reformulates attention as a stochastic process: query-key compatibility scores form an intensity field over a continuous (time × channel) space, sampled via an inhomogeneous Poisson random measure. Although individual outputs are random, the expectation equals a mollified cosine-kernel attention, so the layer directly replaces softmax attention and trains with exact gradients. Crucially, two quantities — evidence (total compatibility mass Λ_q) and disagreement (trace of the value covariance tr Σ_V(q)) — combine in a closed-form variance identity to yield predictive uncertainty in a single deterministic forward pass, with no MC sampling or extra heads. Experiments on t-PatchGNN for irregularly-sampled time series show at most 5.6% accuracy loss (zero on the sparsest dataset), a free disagreement signal beating 20-pass MC Dropout, zero-shot CRPS beating a 50-sample sampler, ranking of 3,383 unseen ICU patients in 1.4 seconds, and nominal coverage with a split-conformal wrapper. The post argues this marks a shift toward embedding uncertainty quantification into the model's core math rather than bolting it on afterward.

Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention

> *Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention* > arXiv:2608.19171 | Sotirios P. Chatzis, Loukas Papadoulas

The Problem: Silent Predictions

Most deep learning models output only a point prediction — a number with no information about how much it should be trusted. In real applications (ICU vital-sign forecasting, market prediction), knowing that a model is uncertain can be more valuable than the prediction itself.

Existing approaches to uncertainty quantification have drawbacks:

Ensembles: train multiple models and compare predictions — prohibitively expensive for large networks.
Bayesian methods: variational inference or MCMC over weights — mathematically elegant but computationally heavy; exact Bayesian inference for large neural nets remains open.
Approximations: MC Dropout, deep ensembles, temperature scaling — cheaper, but approximate and often overconfident.

A related difficulty: most time-series models (RNNs, Transformers, Mamba) assume regular sampling. Real data — glucose monitors, climate stations — is irregularly sampled. Interpolation assumes smoothness, padding invents data, and missing-token approaches add complexity. Continuous-time modeling is more natural but harder.

The Core Idea: Attention as a Stochastic Process

Lévy Attention replaces deterministic attention with a random construction:

1. Query-key compatibility scores form an intensity function over a continuous (time × channel) index space. 2. An inhomogeneous Poisson random measure samples points under this intensity. 3. The output averages an interpolated value field over the sampled points.

Although a single draw is random, the expectation equals a mollified cosine-kernel attention. So the layer can directly replace softmax attention and trains with exact gradients — no Monte Carlo approximation, no reparameterization tricks.

The name references Paul Lévy: like Lévy processes, this attention can "jump" to distant keys, suiting continuous time where the next observation may appear anywhere.

Closed-Form Uncertainty

The Poisson construction preserves information softmax discards, via two quantities:

Evidence Λ_q: total compatibility mass — how much signal supports the query's attention.
Disagreement tr Σ_V(q): dispersion of the attended values — do they agree?

They combine through an exact variance identity:

\hat{\sigma}(q) = \sqrt{\mathrm{tr}\,\Sigma_V(q) \, \varphi(\Lambda_q)}

This is the predictive root-mean-square deviation — output in a single deterministic forward pass, with no trained head and no repeated runs.

Experimental Results

Experiments swap standard attention for Lévy Attention in t-PatchGNN, a state-of-the-art irregular time-series model:

Accuracy: at most 5.6% loss, and zero loss on the sparsest dataset — uncertainty quantification essentially "for free."
vs. 20-pass MC Dropout: the free disagreement signal outperforms 20 forward passes of MC Dropout under matched 5-seed settings.
vs. 50-sample sampler: zero-shot CRPS beats a 50-sample sampler — one pass beats fifty.
Clinical-scale test: ranking 3,383 unseen patients by trustworthiness completed in 1.4 seconds, enabling real-time clinical decision support.
Coverage: with a split-conformal wrapper, it achieves nominal coverage — statistically honest confidence intervals.

When to Trust Which Signal

An interesting finding: disagreement always carries signal, regardless of data density, while evidence becomes strongly informative only on sparse data — with few observations, their total mass is a key credibility indicator. Lévy Attention switches between the two automatically.

Why It Matters

Lévy Attention demonstrates a new paradigm: embedding uncertainty quantification into the model's fundamental operations rather than adding it afterward:

1. Zero extra compute — no repeated runs, no MC sampling, no variational inference. 2. Exact mathematical guarantees — closed-form variance identity, not heuristics. 3. Plug-and-play — drops into any Transformer's softmax attention layer.

In high-stakes domains (healthcare, autonomous driving, financial risk), an AI system that can honestly say "I'm not sure" is worth more than any confident wrong prediction.

References

Chatzis, S. P., & Papadoulas, L. (2026). Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention. *arXiv preprint arXiv:2608.19171*. https://arxiv.org/abs/2608.19171

--- *Paper digest collected via Papers.Cool*