English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

When Attention Goes Blind: A Numerical Underflow Bug Hidden in ALiBi Positional Encoding

Forum topic · ✨步子哥 · 2026-08-05

Summary

An August 2026 arXiv paper, "When Attention Goes Blind" (arXiv:2608.03994), reveals a numerical bug in ALiBi positional encoding: the linear attention bias grows increasingly negative with token distance until exp(x) in softmax underflows to zero in floating-point precision, making attention heads unable to see distant tokens. Underflow thresholds are approximately -103.27 for fp32 and -92.18 for bf16; heads with steeper slopes go blind first, so different heads fail at different context distances. Controlled experiments on a 148M-parameter decoder show standard metrics like perplexity are barely affected, which explains why the bug went unnoticed, while passkey retrieval performance is significantly hurt. Surprisingly, default ALiBi slopes remain a strong baseline on needle-in-a-haystack retrieval. The paper evaluates four training-time fixes—clamping, robust slopes, log-scaled distances, and soft capping—with log-scaled distances giving the most consistent passkey improvements. Practical guidance: monitor retrieval tasks beyond perplexity, and note that existing ALiBi-based pretrained models may also be affected.

When Attention Goes Blind: A Numerical Bug Hidden in ALiBi Positional Encoding

You trained a language model with ALiBi positional encoding. Context window 8K, training loss normal, standard benchmark scores normal. Everything looks fine.

But your model may already be partially blind on needle-in-a-haystack retrieval tasks—some attention heads simply cannot see distant tokens over long ranges, because floating-point underflow zeroes out their attention weights.

This is not a hypothesis. It is the core finding of the August 2026 arXiv paper "When Attention Goes Blind": ALiBi's linear bias underflows under floating-point precision, turning a swath of attention weights into zero and making heads unable to see faraway tokens.

What ALiBi Is, and Why It Goes Blind

A quick recap of ALiBi. In Transformer attention, every token attends to all other tokens. The original Transformer used sinusoidal position encodings; RoPE later became mainstream. ALiBi, proposed by Press et al. in 2022, is an alternative: instead of adding positional encodings to token embeddings, it adds a linear bias to attention scores—the farther the distance, the larger (more negative) the bias, and the smaller the attention weight.

Intuitively sensible: distant tokens matter less than nearby ones, and linear decay is a simple inductive bias.

The problem lies in the numerical implementation of softmax. Attention weights are computed as:

\[\text{softmax}(A + B)\]

where \(A\) is the attention score and \(B = -m \cdot D\) is the bias (\(m\) is the per-head slope, \(D\) is the distance matrix). When distance \(D\) is large enough, \(B\) becomes a large negative number, pushing \(A + B\) to very negative values. Softmax computes \(e^x\) per element—when \(x\) is negative enough, \(e^x\) underflows to zero.

That is the "blindness": beyond a certain distance threshold, the exponentials for distant tokens become exactly zero, their attention weights are wiped out, and the head cannot see them.

When It Triggers

The paper gives concrete thresholds:

  • fp32 (single precision): underflow threshold \(\tau_{u,\text{fp32}} \approx -103.27\)
  • bf16 (bfloat16): underflow threshold \(\tau_{u,\text{bf16}} \approx -92.18\)
  • ALiBi assigns different slopes to different heads. The steeper the slope, the faster the decay and the earlier underflow triggers. The paper notes: "the head with the steepest slope underflows first."

    This means in multi-head models, heads do not go blind simultaneously—they go blind one by one, from steepest to shallowest slope. Some heads already lose sight of distant tokens before the context reaches 8K, while others last longer. The model appears to work, but part of its attention has become nearsighted.

    How Big Is the Impact

    Using controlled experiments on a 148M-parameter decoder, the paper separated this failure mode from "out-of-context degradation" (another known issue). Conclusions:

    1. Little impact on standard benchmarks. Perplexity and downstream task scores barely move. That is why this bug went unnoticed: standard benchmarks cannot see it.

    2. Large impact on token retrieval. On the passkey retrieval task (finding a password inside a long text), ALiBi blindness significantly hurts performance. This makes sense—retrieval requires heads to see distant tokens, and blind heads cannot.

    3. Yet on needle-in-a-haystack, default ALiBi slopes remain a strong baseline. A counterintuitive finding: despite underflow, the default slope configuration performs well on needle-in-a-haystack. The paper's explanation: the default slopes happen to place underflow at "not-so-critical" distances.

    Four Fixes

    The paper evaluated four training-time mitigation strategies:

    1. Clamping: cap the bias at a maximum value so it cannot grow indefinitely 2. Robust Slopes: redesign the slope sequence to avoid premature underflow 3. Log-scaled Distances: replace the linear bias with logarithmic decay—slower decay with distance, avoiding unbounded push toward negative infinity 4. Soft Capping: replace hard clamping with a smooth function

    Result: log-scaled distances most consistently improved passkey retrieval performance.

    The intuition behind log decay: the linear bias assumes importance halves with every fixed distance increment—too aggressive. A log bias assumes importance halves only when distance grows by an order of magnitude—closer to how relevance actually decays.

    What This Tells Us

    This finding is another instance of the "evaluation blind-spot law": failure modes invisible to standard benchmarks are where bugs hide.

    ALiBi's blindness does not affect perplexity or standard downstream scores, so standard evaluation says "all is well." Only when you specifically test long-range retrieval does the problem surface. It is the same class of problem as Progressive Cramming (99% token accuracy masking 100% generation failure) and TriviaRoomQA (fine within the knowledge boundary, random collapse beyond it): a single metric can mask critical failure modes.

    The deeper cause: ALiBi's linear bias assumes unlimited decay—distance grows without bound, the bias grows without bound negative. But floating-point precision is finite. Whenever a mathematical model assumes infinity and the implementation is finite, a mismatch appears at the boundary. This echoes RoPE's precision issues under bfloat16 (Wang et al. 2025): a crack between the mathematical assumptions of positional encoding and the physical reality of floating point.

    Practical Advice

    The paper offers concrete training guidance:

  • If you use ALiBi, consider switching to log-scaled distance biases
  • When training new models, monitor passkey retrieval, not just perplexity
  • If you deploy existing ALiBi models for long-context applications, some attention heads may already be blind at your working distance
  • Default slopes are okay for needle-in-a-haystack, but check passkey retrieval separately
The paper notes the issue also appears in "ALiBi-based SOTA pretrained models"—this is not only a problem when training new models; released models may be affected too.

An Engineering Lesson

This bug stayed hidden so long because it appears only on specific tasks, at specific distances, in a specific way. The "average scores" of standard benchmarks dilute it.

The lesson for anyone designing a new positional encoding: working fine at 1K context does not mean working fine at 32K. Floating-point precision is not infinite, and your formulas may diverge from the implementation at the boundary.

Test long-range retrieval, not just standard benchmarks. Your model may be more nearsighted than you think.

---

Paper: https://arxiv.org/abs/2608.03994

Code: the paper states "Code will be released upon publication"

Tags

#alibi#positional-encoding#attention#floating-point-underflow#long-context#retrieval#transformers#numerical-stability

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178595033