Attention-Path Fragility: When Model Confidence Rests on Shaky Foundations

> Paper: arXiv:2608.11138 > Code: not yet open-source > Authors: Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon

A Counterintuitive Scenario

Ask a large model: "Who wrote One Hundred Years of Solitude?" It answers: "Gabriel García Márquez" with 99.9% confidence.

Ask it: "Who won the 2026 Nobel Prize in Physics?" It answers with a name and 99.9% confidence.

Both answers share identical confidence, but the first is correct and the second is fabricated. The model itself cannot distinguish these two types of "confidence"—at least not from the traditional output probability distribution.

This paper asks an overlooked question: If a model is confident about an answer, but a small perturbation to its attention pathways collapses the answer—is this "fragile confidence" itself an uncertainty signal?

The answer is yes, and this signal is distinct from traditional output confidence.

The Predicament of Existing Uncertainty Estimation

Uncertainty estimation (UE) for LLMs typically follows four routes:

1. Output entropy: Measure how peaked the output distribution is. Sharper distributions indicate higher confidence. The problem: models are often sharply peaked on wrong answers—they do not know what they do not know. 2. Sampling consistency (self-consistency): Run multiple inferences and check whether answers agree. Expensive, requires many forward passes. 3. Semantic consistency: Compare semantic similarity across multiple samples rather than literal agreement. Better than raw sampling, but still requires multiple samples. 4. Structural perturbation: Add noise or dropout to model internals and check output stability. This is the route of the present paper.

The core insight: traditional methods all inspect the "output side"—what the model says. But what the model says depends on how its attention pathways route information. If the pathways are fragile, even a confident output is deceptively confident.

ASMI: Attention-Subnetwork Mutual Information

The method is called ASMI and works in three steps.

Step 1: Mask Attention Heads

Divide the model's attention heads into subnetworks. Randomly mask subsets (similar to dropout, but only on attention layers). Each masking yields an output distribution.

Step 2: Measure Mutual Information

Use the BALD (Bayesian Active Learning by Disagreement) framework to measure mutual information between subnetwork outputs. Intuition: if the model is confident and correct, masking different subsets should yield similar outputs—high mutual information. If the model is confident but fragile, different subsets give divergent answers—low mutual information.

Step 3: Semantic Agreement Kernel

The key innovation. Traditional BALD compares literally identical outputs, but language models often produce outputs that are "literally different but semantically identical"—for example, "Márquez" and "Gabriel García Márquez" refer to the same answer. ASMI introduces a semantic agreement kernel that measures agreement at the semantic level rather than the literal level.

This design is critical. Without it, ASMI would misjudge semantically identical outputs as inconsistent, thereby overestimating uncertainty.

Core Finding: Not a Repackaging of Output Confidence

The most rigorous part of the paper demonstrates that ASMI provides an independent signal, not a transformation of output confidence.

Finding 1: ASMI Halves Error Rate in the Confident-but-Fragile Zone

On grounded QA (SQuAD, CoQA, BAbIQA), using ASMI as an additional signal in a confidence filter:

• In the confident-but-fragile subset, the error rate drops from baseline to roughly half.
• In the confident-and-robust subset, ASMI neither helps nor hurts.

This means ASMI precisely targets the blind spot of traditional methods—those predictions that "look confident but are about to collapse."

Finding 2: Orthogonality of the Signal

Residualization analysis: first remove the uncertainty explainable by output confidence and entropy, then check whether ASMI can still predict errors. Result: yes. Across five model-dataset combinations, ASMI provided statistically significant incremental information in four.

Finding 3: Has Its Own Operating Envelope

ASMI fails on parametric QA (TriviaQA, closed-book QA)—the signal even flips direction. The authors do not hide this and provide an explanation:

• Grounded QA (with context): Answers must be extracted from context, so attention pathways are critical. Pathway fragility = high uncertainty.
• Parametric QA (closed-book): Answers depend on parametric knowledge, so attention pathways play a different role. Pathway fragility does not equal uncertainty.

This boundary is important. ASMI is not a universal remedy—it has a defined operating envelope: coverage ≤ 0.15, subnetwork size S ≈ 40.

Why This Method Is Interesting

1. "Inspect the Foundation, Not the Signboard"

Traditional UE inspects the "signboard"—the output probability distribution. ASMI inspects the "foundation"—whether the attention pathways are stable. A signboard can be polished, but if the foundation is loose, the building will eventually collapse.

This idea generalizes. Any complex system has a distinction between "surface signals" and "structural signals." Human confidence can be "claiming certainty" (output confidence) or "answering consistently no matter how you are asked" (structural robustness). The latter is more reliable.

2. Training-Free

ASMI does not require retraining or extra labeled data. It only requires multiple masked forward passes at inference. Although more expensive than a single inference (about 40 forward passes), it is cheaper than sampling-consistency methods, which require full answer generation.

3. Another Instance of the "Evaluation Blind Spot Law"

This paper exposes an overlooked failure mode: confident but fragile. Traditional evaluation looks at average accuracy or average confidence calibration and completely misses this subset. This is analogous to a medical checkup that only looks at average blood pressure but misses patients with "normal blood pressure yet low heart-rate variability."

This resonates with several prior observations:

• Progressive Cramming: 99% token accuracy masks 100% generation failure.
• Token Budget: a 96.5% vs 11.5% bimodal fate encoded in representations from early layers.
• QuantiBias: quantization introduces bias in the blind spots of standard safety checks.

Single metrics mask critical failure modes—this principle grows increasingly robust.

Honest Limitations

The paper does not shy away from limitations:

1. Computational cost: 40 forward passes are unfriendly for real-time deployment. The paper offers an approximation (Top-K masking) but with reduced accuracy. 2. Only attention layers tested: MLP layers, residual streams, and other components are not involved. Attention-path fragility may be only the tip of the iceberg. 3. Narrow applicability: Effective only on grounded QA, fails on parametric QA. ASMI cannot serve as a universal uncertainty estimator; the choice depends on task type. 4. Model scale: Primarily tested on 1B-8B models. Larger models may exhibit different attention-pathway behavior—more robust or more fragile is currently unclear.

Implications for Engineering Practice

If you are building a RAG system (context-grounded QA), ASMI is worth considering:

• High-stakes deployment: medical, legal, financial QA, where error cost far exceeds the cost of 40 extra forward passes.
• Confidence routing in RAG systems: use ASMI to decide which queries need retrieval augmentation, which can be answered directly, and which require human review.
• Model selection: compare models on the confident-but-fragile subset—this reflects reliability better than average accuracy.

For closed-book QA, creative writing, or dialogue systems, ASMI currently offers limited help.

One-Sentence Summary

Model confidence comes in two forms: robust confidence that survives perturbation, and fragile confidence that shatters at the slightest touch. ASMI is the first training-free method that distinguishes these two at the attention-pathway level—it does not replace traditional confidence but complements the half of the world traditional confidence cannot see.

---

> Paper: Attention-Path Fragility as an Uncertainty Signal in Large Language Models > Authors: Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon > Date: 2026-08-11