English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

When Attention Goes Blind: A Floating-Point Trap Hidden in ALiBi Positional Encodings for Three Years

Forum topic · ✨步子哥 · 2026-08-05

Summary

A 2026 study by Christopher Schröder's team (InfAI / ScaDS.AI / University of Kassel / hessian.AI, arXiv:2608.03994) reveals that ALiBi positional encodings cause complete, silent attention failure in popular LLMs. Because ALiBi adds a linear bias (-m·D) to attention scores, the softmax exponentials eventually underflow in finite-precision arithmetic: below roughly -92.18 in bf16 (or -103.27 in fp32), e^x becomes exactly zero, turning gradual attention decay into hard truncation. Tests on BLOOM-560M, Falcon-RW-7B, and MPT-7B show some heads go blind well before the 2048-token context limit — Falcon's steepest heads under 1,000 tokens. Standard benchmarks (CommonSense, QA, language generation) miss the problem entirely, with only 1.6–3.4 points difference between mitigation strategies, while passkey retrieval exposes it: baseline ALiBi scores AUC 0.08 versus 0.79 for a Clamping + Log-scaled combination — a ~10x improvement. The paper proposes four training-time mitigations (Clamping, Explicit, Log-scaled bias, Softcap), notes none wins everywhere, and finds default ALiBi slopes remain a surprisingly strong baseline on needle-in-a-haystack tasks. The deeper insight is a level mismatch between idealized math and finite-precision implementations, with practical advice for practitioners on precision, bias functions, and evaluation probes.

The Invisible Blind Spot

Suppose you train a 7B-parameter language model with a 2048-token context window, using ALiBi positional encoding — it's cheap, parameter-free, and extrapolates to longer contexts. Standard evaluations look fine, CS/QA/LG benchmarks differ by only 1.6 to 3.4 percentage points, and you release the model.

Then someone puts the sentence "the passkey is 42-7B" 3,000 tokens earlier into your model's context and asks what the passkey is. The model can't answer.

It's not stupid. Its attention heads, starting from token 2048, simply cannot see the passkey anymore. Not blurred — completely invisible: the corresponding attention weights are zeroed out by floating-point precision. Your model doesn't know it's blind, and neither do you. Standard benchmarks can't detect it.

This is "ALiBi blindness," discovered in August 2026 by Christopher Schröder's team at the University of Leipzig (arXiv:2608.03994). BLOOM 560M, Falcon-RW 7B, and MPT 7B — every SOTA pretrained model using ALiBi — is affected.

ALiBi's Design Intent vs. Mathematical Reality

ALiBi (Attention with Linear Biases), proposed by Press et al. in 2022, skips positional embeddings and instead adds a linear bias to attention scores — the farther apart tokens are, the larger the bias and the smaller the attention weight:

\[B = -m \cdot D\]

where \(m\) is each head's slope and \(D\) is the token-pair distance. Slopes are assigned in a geometric progression: steep heads focus near, flat heads focus far. The design intent: smooth attention transitions and extrapolation to longer contexts.

The design is elegant. But floating-point math disagrees.

Softmax computes \(e^{x_i}\) at every step. As \(x_i\) grows more negative (because the bias \(-m \cdot D\) grows linearly with distance), the exponential eventually falls below the smallest positive number representable in floating point. The underflow threshold is \(\tau_{u, \text{bf16}} \approx -92.18\) for bf16 and \(-103.27\) for fp32. Past that point, \(e^{x_i}\) becomes exactly 0.

Attention weight goes from "very small" to "zero." Not decay — truncation.

The Mechanism: From Gradient to Black Screen

ALiBi's linear bias is like gradient-tinted glasses: distant things look dimmer, nearby things clear. The designer hoped the wearer could see anything at any distance, just slightly blurred far away.

But floating-point precision is a hard material limit of the lens. At sufficient distance, \(e^{x_i}\) goes to zero — the lens switches from "gradient" to "completely opaque." The wearer doesn't know they can't see, because those objects aren't blurry — they don't exist.

Worse, ALiBi heads have different slopes. Steep heads go blind first, flat heads later — what the paper calls "partial" and "complete" blindness. In a 16-head layer under bf16, the steepest heads blind out within a few hundred tokens while the flattest survive to several thousand. The layer looks functional — not all heads are blind — but long-range attention has been quietly hollowed out.

Three SOTA Models, No Exceptions

| Model | Params | Context | Layers | Heads | Slope range | |-------|--------|---------|--------|-------|-------------| | BLOOM | 560M | 2048 | 24 | 16 | \(2^{-0.5}\) to \(2^{-8}\) | | Falcon-RW | 7B | 2048 | 36 | 64 | \(2^{-0.125}\) to \(2^{-8}\) | | MPT | 7B | 2048 | 32 | 32 | \(2^{-0.25}\) to \(2^{-8}\) |

All three use the original geometric-progression slopes. MPT adds clamping (\(\delta_1\)) as a partial mitigation; the other two run completely unprotected.

Perplexity probes show that in all models, some heads begin underflowing before reaching the 2048-token context limit. Falcon-RW 7B's steepest heads go blind under 1,000 tokens; BLOOM 560M even earlier.

The Evaluation Blind Spot: Why Nobody Noticed for Three Years

The chilling part: blindness existed for three years across BLOOM, Falcon, and MPT, yet standard benchmarks completely miss it.

On the CS (CommonSense), QA, and LG (Language Generation) benchmarks, differences between mitigation strategies are only 1.6–3.4 points. Even deliberately making models "more blind" barely moves these scores.

But passkey retrieval and Needle-In-a-HayStack (NIHS) — both requiring the model to retrieve a specific token from long context — are different:

  • Passkey retrieval (out-of-context): ALiBi baseline AUC 0.08, nearly total failure. Clamping + Log-scaled combination: 0.79 — a nearly 10x improvement.
  • NIHS: default ALiBi slopes hold up surprisingly well, because the needle sits relatively close to the prompt.
This is a textbook case of the "evaluation blind spot law": models behave normally on whatever you measure; the problem hides where you don't. Standard benchmarks measure generation, not retrieval. Blind heads don't hurt generation, which relies on local continuity — but retrieval demands precision at distance, and one blind head is fatal.

This echoes a recurring theme: measurement coverage matters more than measurement depth.

Four Mitigations: From Patch to Level Shift

The paper proposes four training-time mitigations:

1. Clamping (C): floor the bias at a safe threshold, \(B^* = \max(B, c_{\text{clamp}})\). Beyond the threshold, all distances look alike — position information is lost, but underflow is prevented. 2. Explicit (E): explicitly set underflowed attention weights to a tiny nonzero value. 3. Log-scaled (L): replace the linear bias with a logarithmic one, \(B = -m \cdot \log(D+1)\). Bias grows ever more slowly with distance and never triggers underflow. 4. Softcap (S): compress the bias into a range with a softcap function.

Individually, Log-scaled is most stable on passkey retrieval (AUC 0.77); Clamping alone reaches only 0.08. Combined: Clamping + Log-scaled = 0.79, ~10x over baseline.

A counterintuitive finding: no strategy wins on every task. Log-scaled wins on passkey but is slightly worse on standard benchmarks. Clamping is okay on NIHS but nearly useless on passkey. Combinations interfere — C+S is worse than C alone.

The paper's surprising conclusion: the default ALiBi slopes remain a surprisingly strong baseline, especially on NIHS. Possibly the geometric progression keeps most heads' slopes flat (no underflow), while the steepest blind heads weren't responsible for long-range attention anyway.

Engineering Insight: Not a Bug, but a Level Mismatch

The deepest insight isn't "ALiBi has a bug" — it's a level mismatch between formal definitions and finite-precision implementations.

ALiBi's mathematical definition assumes real arithmetic: the linear bias can grow unboundedly and softmax never truly zeroes out. Actual training runs bf16, whose smallest positive number corresponds to an exponent around \(-126\). Below that: zero.

This isn't unique to ALiBi. RoPE has similar numerical failures: Wang et al. (2025) showed RoPE's relative position encoding distorts under bf16; Gemma's low-frequency rotation signals weaken at long context; Du et al. (2026) found RoPE loses its ability to distinguish positions and tokens in long contexts.

Every positional encoding has its own failure mode under finite precision. ALiBi's is the stealthiest, hiding inside softmax's exponential rather than the encoding itself.

The fix exemplifies "solving at a different level": not "make ALiBi more precise," but swap in a functional form that can't trigger underflow. Log-scaled bias isn't a more precise linear bias — it's a different level, replacing linear growth with logarithmic growth.

Practical Recommendations

1. Check your precision: blindness starts earliest in bf16, much later in fp32. If retrieval matters, consider fp32 or mixed precision. 2. Use log-scaled bias: when training new models, replace \(B = -m \cdot D\) with \(B = -m \cdot \log(D+1)\). Near-zero cost; ~10x passkey retrieval improvement. 3. Add clamping: floor the bias at inference time to prevent underflow. Crude but effective. 4. Don't trust standard benchmarks alone: CS/QA/LG can't detect this. If your use case involves long-context retrieval (RAG, long-document QA, code understanding), add passkey or NIHS probes. 5. No retraining needed for existing models: clamping can be applied at inference directly. Log-scaled requires retraining but works best.

Closing Thought: Overlooked Insights Are More Dangerous Than Novel Discoveries

ALiBi was proposed in 2022, adopted by three SOTA models, and cited by countless papers — yet its numerical failure mode was only systematically exposed in 2026. Why? Standard benchmarks can't detect it; blind heads don't hurt generation; and the promise that "ALiBi extrapolates" was too beautiful for anyone to verify numerically.

More fundamentally, the paper exposes the gap between mathematical definitions and engineering implementations. Formulas assume reals; code runs floats. This gap is everywhere in AI — softmax instability, gradient explosion/vanishing, fp16 loss scaling — but rarely studied systematically as a *design* problem. ALiBi blindness may be the first complete, quantitative, mitigation-equipped treatment of it.

For anyone building agent systems there's an extra warning: an agent's long-context retrieval may be silently capped by its positional encoding. If your agent is built on Falcon or MPT, retrieving key earlier information in long conversations may simply fail — not because the model is dumb, but because its attention heads are blind. RoPE-based models (Llama, Qwen) may fare better, but RoPE has its own numerical failure modes. In the agent era, the numerical robustness of positional encoding is an underrated engineering dimension.

---

Paper: When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings Authors: Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer Affiliations: InfAI / ScaDS.AI Dresden/Leipzig / University of Kassel / hessian.AI arXiv: https://arxiv.org/abs/2608.03994 HTML full text: https://arxiv.org/html/2608.03994v1 Code: to be released upon publication Related tooling: lm-evaluation-harness (https://github.com/EleutherAI/lm-evaluation-harness), needle-in-a-haystack (https://github.com/gkamradt/needle-in-a-haystack)

Tags

#alibi#positional-encoding#floating-point#llm#attention#long-context#bf16#evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178595034