Attention-Path Fragility as an Uncertainty Signal in LLMs: An Introduction to ASMI

> Paper: arXiv:2608.11138 > Code: not yet open-sourced > Authors: Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon

A Counterintuitive Scenario

Ask a large language model: "Who wrote *One Hundred Years of Solitude*?" It answers "Gabriel García Márquez" with 99.9% confidence.

Ask instead: "Who won the 2026 Nobel Prize in Physics?" It answers with some name—and the same 99.9% confidence.

The confidence scores are identical, but the first answer is correct and the second is fabricated. The model itself cannot distinguish these two kinds of "confidence"—at least not from the traditional output probability distribution.

The paper asks a neglected question: if a model is confident about an answer, but the answer collapses under slight perturbations to its attention pathways, is this "fragile confidence" itself an uncertainty signal?

The answer is yes—and the signal is distinct from traditional output confidence.

The Dilemma of Existing Uncertainty Estimation

Uncertainty estimation (UE) for LLMs follows four main routes:

1. Output entropy: measure how spread out the output distribution is. Sharper distributions suggest more confidence—but models are often sharp on wrong answers too. They don't know what they don't know. 2. Self-consistency: sample multiple answers and check agreement. Costly—many forward passes. 3. Semantic consistency: compare semantic similarity across samples rather than literal match. Better, but still requires multiple samples. 4. Structural perturbation: inject noise or dropout into the model's internals and check output stability. This paper belongs to this route.

The core insight: traditional methods look at the "output end"—what the model says. But what a model says depends on how its attention pathways are routed. If those pathways are fragile—slight perturbation changes the answer—then even confident output is false confidence.

ASMI: Attention-Subnetwork Mutual Information

The method, called ASMI (Attention-Subnetwork Mutual Information), works in three steps:

Step 1: Mask Attention Heads

Divide the model's attention heads into several subnetworks and randomly mask a portion of them each time (like dropout, but only in attention layers). Each masking produces an output distribution.

Step 2: Measure Mutual Information

Using the BALD (Bayesian Active Learning by Disagreement) framework, measure mutual information across subnetwork outputs. Intuition: if a model is confident and correct, masking different attention heads should barely change the output—high mutual information. If the model is confident but fragile, different masked subsets yield wildly different answers—low mutual information.

Step 3: Semantic Agreement Kernel

The key innovation. Traditional BALD compares outputs literally, but LLM outputs are often "literally different yet semantically identical"—"Márquez" and "Gabriel García Márquez" are the same answer. ASMI introduces a semantic agreement kernel that measures agreement between subnetworks by semantic equivalence rather than literal match.

This design is crucial. Without it, ASMI would misjudge "semantically same, literally different" outputs as disagreement, overestimating uncertainty.

Core Findings: Not an Output-Confidence Echo

The paper's most solid contribution is proving ASMI provides an independent signal, not a re-skinned output confidence.

Finding 1: In the "Confident Yet Fragile" Region, ASMI Cuts Error Rates in Half

On grounded QA (SQuAD, CoQA, BabiQA), using ASMI as an additive confidence-filtering signal:

• In the subset of "confident but fragile" predictions, error rates dropped to roughly half of baseline
• In the "confident and robust" subset, ASMI added nothing and removed nothing

This means ASMI precisely targets the blind spot of traditional methods—predictions that "look confident but are about to collapse."

Finding 2: Signal Orthogonality

Using residualization analysis—first removing the uncertainty explainable by output confidence and entropy, then testing whether ASMI still predicts errors—the answer is yes. Across five model-dataset combinations, ASMI provided statistically significant incremental information in four.

Finding 3: It Has Its Own Boundary of Applicability

ASMI fails on parametric QA (TriviaQA, closed-book)—the signal direction even inverts. The authors don't hide this; they explain it:

• Grounded QA (with context): the answer must be extracted from context, so attention pathways are critical. Fragile pathways = high uncertainty.
• Parametric QA (closed-book): the answer relies on knowledge in the model's parameters, and attention pathways play a different role. Fragile pathways do not equal uncertainty.

This boundary matters. ASMI is not a panacea—it has its own "operating envelope." The paper even gives quantitative envelope conditions: coverage ≤ 0.15, subnetwork size S ≈ 40.

Why This Method Is Interesting

1. "Check the Foundation, Not the Signboard"

Traditional uncertainty estimation looks at the model's "signboard"—the output probability distribution. ASMI looks at the "foundation"—whether the attention pathways are stable. A signboard can be polished to a shine, but if the foundation is loose, the building will eventually collapse.

This idea generalizes. Any complex system distinguishes "surface signals" from "structural signals." Human confidence can be "saying you're sure" (output confidence) or "being able to answer no matter how you're asked" (structural robustness). The latter is more reliable.

2. Training-Free

ASMI requires no retraining and no additional labeled data—only multiple masked forward passes at inference. Though more expensive than a single pass (~40 forward passes), it's cheaper than sampling-consistency methods, which must generate complete answers.

3. Another Case for the "Evaluation Blind-Spot Law"

The paper reveals a neglected failure mode: confident yet fragile. Traditional evaluation only sees "average accuracy" or "average confidence calibration" and completely misses this subset—like a medical check that only reads average blood pressure and misses high-risk patients with normal blood pressure but low heart-rate variability.

This resonates with previously observed papers:

• Progressive Cramming: 99% token accuracy masking 100% generation failure
• Token Budget: a bimodal fate (96.5% vs 11.5%) encoded early in representations
• QuantiBias: quantization introducing bias in the blind spots of standard safety checks

Single metrics hide critical failure modes—this law keeps solidifying.

Honest Limitations

The paper does not avoid its limitations:

1. Compute overhead: 40 forward passes is unfriendly to real-time deployment. The paper offers an approximation (Top-K masking), with some accuracy loss. 2. Attention layers only: MLP layers and other components of the residual stream are untouched. Attention-path fragility may be just the tip of the iceberg. 3. Narrow applicability: effective only on grounded QA, failing on parametric QA. ASMI cannot serve as a universal uncertainty estimator; task type matters. 4. Model scale: tested mainly on 1B–8B models. How attention pathways behave in much larger models—more robust or more fragile—remains unclear.

Implications for Engineering Practice

If you're building a RAG system (context-grounded QA), ASMI is worth considering. Concrete scenarios:

• High-stakes deployment: medical, legal, financial QA, where the cost of errors far exceeds the cost of 40 extra forward passes
• Confidence routing for RAG: use ASMI to decide which queries need retrieval augmentation, which can be answered directly, and which need human review
• Model selection: compare models on the "confident yet fragile" subset—more revealing of reliability than average accuracy

But for closed-book QA, creative writing, or dialogue systems, ASMI currently offers limited help.

One-Sentence Summary

A model's confidence comes in two kinds: robust confidence that survives perturbation, and fragile confidence that shatters at a touch. ASMI is the first training-free method to distinguish the two at the attention-pathway level—it doesn't replace traditional confidence, it completes the half of the world traditional confidence cannot see.

---

> Paper: Attention-Path Fragility as an Uncertainty Signal in Large Language Models > Authors: Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon > Published: 2026-08-11