Hidden Numerical Bug in ALiBi: When Attention Goes Blind
You train a language model with ALiBi positional encoding. Context window 8K. Training loss looks fine. Standard benchmark scores look fine. You assume everything works.
But your model may already be partially blind on needle-in-a-haystack retrieval: certain attention heads cannot see distant tokens because floating-point underflow has zeroed out their attention weights.
This is not speculation. It is the core finding of the August 2026 arXiv paper "When Attention Goes Blind": ALiBi's linear bias underflows in floating point, turning large slices of the attention weights into zero and making heads "invisible" to distant tokens.
What ALiBi Is and Why It Goes Blind
ALiBi, proposed by Press et al. in 2022, skips positional embeddings in token vectors and instead adds a linear bias to attention scores. The further away a token is, the more negative the bias, and the smaller the resulting attention weight. The intuition: distant tokens matter less than nearby ones, so apply a linear decay as a simple inductive bias.
The problem shows up in softmax's numerical implementation:
$$\text{softmax}(A + B)$$
where $A$ is the attention score and $B = -m \cdot D$ is the bias ($m$ is the per-head slope, $D$ the distance matrix). When $D$ is large enough, $B$ becomes a large negative number, pushing $A + B$ far below zero. Softmax needs $e^x$ for every element. When $x$ is negative enough, $e^x$ underflows to zero.
That is the blindness: past a distance threshold, distant tokens' exponentials collapse to zero, their attention weight is wiped out, and the head can no longer see them.
When It Triggers
The paper gives concrete thresholds:
- fp32: underflow threshold $\tau_{u,\text{fp32}} \approx -103.27$
- bf16: underflow threshold $\tau_{u,\text{bf16}} \approx -92.18$
- If you use ALiBi, consider switching to log-scaled distances.
- If you train a new model, monitor passkey retrieval, not only perplexity.
- If you deploy an existing ALiBi model for long-context use, check whether some heads are already blind at your operating distance.
- Default slopes are acceptable on NIAH, but passkey retrieval needs separate verification.
Different ALiBi heads use different slopes. The steeper the slope, the faster the decay, and the earlier that head underflows. The paper notes: "the head with the steepest slope underflows first."
In multi-head models, heads do not all go blind at once. They go blind in order, steepest slope first. Some heads are nearsighted well before 8K; others hold out much longer. The model still appears to function, but a fraction of its heads have effectively become nearsighted.
How Bad Is the Impact
Using a controlled 148M-parameter decoder-only model, the paper separates this failure mode from out-of-context degradation. Three findings stand out:
1. Standard benchmarks barely move. Perplexity and downstream scores stay nearly flat. That is why the bug went unnoticed: standard evaluation cannot see it.
2. Token retrieval suffers badly. On passkey retrieval (finding a password hidden in a long context), the blindness meaningfully hurts performance. This is intuitive: retrieval needs heads to see distant tokens, and blind heads cannot deliver.
3. Yet default ALiBi slopes are still a strong baseline on needle-in-a-haystack. A counter-intuitive result: despite the underflow problem, the default slope schedule still does well on NIAH retrieval. The paper's explanation is that the defaults happen to push underflow into a distance range that is not critical for that task.
Four Fixes Tested
The paper evaluates four training-time mitigations:
1. Clamping: cap the bias at a maximum magnitude so it cannot run off to $-\infty$. 2. Robust Slopes: redesign the slope schedule to slow underflow. 3. Log-scaled Distances: replace linear bias with log decay, so the bias no longer has an unbounded negative tail. 4. Soft Capping: use a smooth function instead of a hard cap.
Result: Log-scaled distances give the most consistent improvement on passkey retrieval.
The intuition: a linear bias encodes "distance doubles, importance halves." That is too aggressive. A log bias encodes "distance grows by an order of magnitude, importance halves," which matches how relevance actually fades in real contexts.
What This Says About Evaluation
This is another instance of the "evaluation blind spot law": the failure modes standard benchmarks cannot see are exactly where bugs hide.
ALiBi's blindness does not move perplexity or standard downstream scores, so standard evals say "all clear." Probe long-range retrieval specifically, and the problem surfaces. This joins other examples like Progressive Cramming (99% token accuracy masking 100% generation failure) and TriviaRoomQA (in-distribution OK, out-of-distribution random) as cases where single-metric reporting hides critical failures.
The deeper cause: ALiBi's linear bias assumes unbounded decay. Distance can grow without limit; bias can grow without bound toward $-\infty$. But floating point has finite precision. When the math says "infinite" and the hardware says "finite," the seam shows. This is the same class of issue as RoPE precision problems in bfloat16 (Wang et al. 2025): the mathematical assumptions of a positional encoding and the physical reality of floating point do not quite agree at the boundary.
Practical Recommendations
The paper's concrete advice for trainers:
An Engineering Lesson
This bug hid for so long because it appears only on specific tasks, at specific distances, in specific ways. Standard "average score" reporting dilutes it away.
The lesson for anyone designing a new positional encoding: working at 1K context does not guarantee working at 32K. Floating-point precision is not infinite. Your formula may diverge from its implementation right at the boundary.
Test long-range retrieval. Do not rely only on standard benchmarks. Your model may be more nearsighted than you think.
---
Paper: https://arxiv.org/abs/2608.03994
Code: The paper states "Code will be released upon publication."