An Invisible Blind Spot
Imagine training a 7B-parameter language model with a 2048-token context window using ALiBi positional encoding—chosen for its zero-parameter overhead and length extrapolation promises. Standard benchmarks pass with perplexity in line and downstream scores within 1.6 to 3.4 percentage points of baselines. You release the model.
Then someone inserts "the password is 42-7B" 3000 tokens before the question. The model cannot retrieve it.
The model is not stupid. Its attention heads have effectively gone blind beyond token 2048: the corresponding attention weights have underflowed to zero in floating-point arithmetic. The model does not know it cannot see, you do not know, and standard benchmarks never detected it.
This is the ALiBi blindness identified by Christopher Schröder's team at Leipzig University (arXiv:2608.03994). BLOOM 560M, Falcon-RW 7B, and MPT 7B—all SOTA pretrained models using ALiBi—are affected.
Design Intent vs. Mathematical Reality
ALiBi (Attention with Linear Biases), introduced by Press et al. in 2022, adds a linear bias to attention scores instead of position embeddings:
$$B = -m \cdot D$$
where m is a per-head slope and D is the token-pair distance. Slopes are assigned geometrically: steep heads focus nearby, flat heads attend far away. The intent is smooth distance decay plus extrapolation to longer contexts.
The design is elegant, but floating-point arithmetic disagrees.
Softmax computes e^{x_i}. As x_i grows increasingly negative (due to -m · D), the exponential eventually underflows. bf16 underflows at τ_u,bf16 ≈ -92.18; fp32 at -103.27. Below this threshold, e^{x_i} becomes exactly 0. Attention weights transition from "small" to zero—not decay, but truncation.
The Mechanism: From Gradual Fade to Blackout
Consider the linear bias as gradient glasses: distant objects appear dimmer, nearby ones clear. The designer intended the wearer to perceive any distance, just slightly blurred when far.
But float precision is the material limit. When distance grows large enough, e^{x_i} underflows to zero—the lens shifts from "gradient" to "fully opaque." The wearer does not know they cannot see, because those objects truly do not exist in their field of view.
Worse, ALiBi heads have different slopes. Steep heads blind first; flat heads blind last. The paper calls these "partial blindness" and "full blindness." In a 16-head layer at bf16, the steepest head loses vision within hundreds of tokens, the flattest holds out for thousands. The layer appears functional—not every head is blind—but long-range attention capacity has been silently hollowed out.
Three SOTA Models, All Affected
| Model | Parameters | Context | Layers | Heads | Slope Range | |-------|-----------|---------|--------|-------|-------------| | BLOOM | 560M | 2048 | 24 | 16 | 2^-0.5 to 2^-8 | | Falcon-RW | 7B | 2048 | 36 | 64 | 2^-0.125 to 2^-8 | | MPT | 7B | 2048 | 32 | 32 | 2^-0.25 to 2^-8 |
All three use the original Press et al. geometric slopes. MPT adds a clamping limit δ₁; the other two run unprotected.
Perplexity probes reveal: some heads in all models underflow before context 2048. Falcon-RW 7B's steepest heads blind before 1000 tokens. BLOOM 560M earlier still.
The Evaluation Blind Spot: Why It Took Three Years
This is the paper's most chilling finding. Blindness existed for three years across BLOOM, Falcon, and MPT, yet standard benchmarks could not detect it.
CommonSense (CS), QA, and Language Generation (LG) benchmarks were tested. Mitigation strategies differed by only 1.6 to 3.4 percentage points. Even deliberately making models "more blind" barely registered.
Passkey retrieval and Needle-In-a-HayStack (NIHS) tell a different story:
- Passkey retrieval (out-of-context): ALiBi baseline AUC 0.08—near-total failure. Clamping + Log-scaled reached 0.79, a ~10× improvement.
- NIHS: Default ALiBi slopes performed surprisingly well, since the target token sits near the prompt.
Four Mitigation Strategies: From Patches to Layer Changes
The paper proposes four training-time mitigations:
1. Clamping (C): Truncate bias to a safe threshold. B* = max(B, c_clamp). Beyond the threshold all distances are treated equally—position info is lost, but underflow is prevented.
2. Explicit (E): Explicitly set underflowing weights to a tiny non-zero value.
3. Log-scaled (L): Replace linear bias with logarithmic bias. B = -m · log(D+1). Growth slows with distance, never triggering underflow.
4. Softcap (S): Apply a softcap function to constrain the bias range.
Standalone results on passkey: Log-scaled most stable at AUC 0.77; Clamping 0.08. Combined Clamping + Log-scaled reaches 0.79, ten times the baseline.
A counterintuitive finding: no single strategy wins everywhere. Log-scaled wins passkey but slightly loses on standard benchmarks. Clamping helps NIHS but barely helps passkey. Combinations interfere—C+S performs worse than C alone.
The paper concludes with a surprise: the default ALiBi slope remains a strong baseline, especially on NIHS. Geometric progression keeps most heads gentle enough to avoid underflow; the few steep heads that go blind were never responsible for long-range attention.
Engineering Insight: This Is Not a Bug, It Is a Layer Mismatch
The deepest insight is not "ALiBi has a bug" but a layer mismatch between formal definition and finite-precision implementation.
ALiBi's math assumes real-number arithmetic: the linear bias can grow without bound, softmax never truly reaches zero. Actual training uses bf16, where the smallest positive normal is 2^-126 (exponent -126). Anything smaller is zero.
This is not unique to ALiBi. RoPE has similar numerical failures: Wang et al. (2025) showed bf16 relative position encoding distorts in RoPE; Gemma's low-frequency rotational signals weaken at long context; Du et al. (2026) found RoPE loses position-token distinguishability at length.
Every positional encoding has finite-precision failure modes. ALiBi's failure is most insidious because it hides inside softmax's exponential, not the encoding itself.
The fix is not "make ALiBi more precise" but "switch to a function form that does not trigger underflow". Log-scaled bias is not a more accurate linear bias—it is another layer: replacing linear growth with logarithmic growth, sidestepping the trap entirely.
Practical Recommendations
1. Check your precision: ALiBi blinds earliest in bf16, latest in fp32. For retrieval-critical workloads, consider fp32 or mixed precision.
2. Use Log-scaled bias: When training new models, replace B = -m · D with B = -m · log(D+1). Near-zero cost, ~10× passkey retrieval improvement.
3. Add Clamping: At inference time, set a lower bound on bias to prevent underflow. Simple and effective.
4. Do not trust standard benchmarks alone: CS/QA/LG cannot detect this issue. For RAG, long-document QA, or code understanding, add passkey or NIHS probes.
5. Existing models need no retraining: Clamping applies at inference. Log-scaled requires retraining but yields the best results.
Broader Reflection
This paper echoes a recurring theme: major breakthroughs are not new tools but renewed scrutiny of existing ones. ALiBi was proposed in 2022, adopted by three SOTA models, and cited in countless papers—yet its numerical failure remained unexposed until 2026.
Why three years undetected? Standard benchmarks could not see it. Blinded heads did not hurt generation quality. The promise "ALiBi extrapolates" was too beautiful to question numerically.
This mirrors other AI evaluation blind spots: Progressive Cramming's 99% token accuracy masking 100% generation failure; Looping Is Not Reliability's 82% accuracy masking 16% rollback; Regression Tax's 59% gains canceled by regressions. Single-metric evaluation masks critical failure modes—this is a law-level regularity.
Deeper still: the paper exposes the gap between mathematical definition and engineering implementation. Formulas assume real numbers; code runs on floats. This gap is pervasive in AI—softmax numerical instability, gradient explosion, gradient vanishing, fp16 loss scaling—but is rarely studied systematically as a "design problem." ALiBi blindness is the first quantitative, reproducible, mitigation-complete disclosure.
For Agent builders specifically: long-context retrieval in agents may be silently limited by positional encoding. If your agent stacks on Falcon or MPT, retrieving early-turn critical information may fail—not from model stupidity, but from attention blindness. Switching to RoPE models (Llama, Qwen) helps somewhat, but RoPE has its own numerical failures. In the agent era, numerical robustness of positional encoding is an underappreciated engineering dimension.
---
Paper: When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings Authors: Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer Institutions: InfAI / ScaDS.AI Dresden/Leipzig / University of Kassel / hessian.AI arXiv: https://arxiv.org/abs/2608.03994 HTML: https://arxiv.org/html/2608.03994v1 Code: To be released upon publication Related evaluation tools: lm-evaluation-harness, needle-in-a-haystack