Introduction
A forum post on zhichai.net discusses a recent paper exploring a counterintuitive question: if a model is confident about an answer, but the answer collapses under slight perturbations to its attention pathways, is that "fragile confidence" itself an uncertainty signal?
The motivating example: ask a model who wrote *One Hundred Years of Solitude* and who won the 2026 Nobel Prize in Physics — both answers come back with 99.9% confidence, but one is correct and one is hallucinated. Traditional output probability distributions cannot distinguish the two kinds of confidence.
The paper: Attention-Path Fragility as an Uncertainty Signal in Large Language Models by Minsoo Kim, Sungyoung Ji, Kisung Moon, and Ilyong Yoon (no public code yet).
Background: Existing Uncertainty Estimation
The post outlines four main approaches to LLM uncertainty estimation (UE):
1. Output entropy — sharp output distributions suggest confidence, but models are often sharp on wrong answers too. 2. Self-consistency — sample multiple answers and check agreement; expensive (multiple full generations). 3. Semantic consistency — compare semantic similarity across samples; better, still requires multiple samples. 4. Structural perturbation — inject noise/dropout internally and check output stability; this is the paper's路线.
Key insight: traditional methods look at the output side. But outputs depend on how attention pathways are routed. If a pathway is fragile — small perturbations change the answer — then confidence is spurious even if the output looks sharp.
The ASMI Method
ASMI (Attention-Subnetwork Mutual Information) works in three steps:
1. Mask attention heads: partition heads into subnetworks and randomly mask subsets (dropout restricted to attention layers), producing an output distribution per mask. 2. Measure mutual information: using the BALD (Bayesian Active Learning by Disagreement) framework. If the model is genuinely confident, masking different subsets yields similar outputs (high mutual information / agreement). If confidence is fragile, different subsets give divergent answers. 3. Semantic agreement kernel: the key innovation. Literal string comparison would treat "Marquez" and "Gabriel Garcia Marquez" as disagreement. The semantic kernel counts semantically equivalent outputs as agreement, avoiding overestimation of uncertainty.
Key Findings
- Halved error in the "confident but fragile" subset: On grounded QA (SQuAD, CoQA, BabiQA), adding ASMI as a confidence filter reduced the error rate to roughly half in the confident-but-fragile subset, with no effect in the confident-and-robust subset.
- Orthogonality to output confidence: Residualization analysis (removing variance explained by output confidence and entropy) shows ASMI still predicts errors — statistically significant in four of five model-dataset pairs.
- A clear applicability boundary: ASMI fails on parametric QA (TriviaQA, closed-book), where the signal direction even reverses. The authors' explanation: in grounded QA, answers are extracted from context, so attention pathways are critical; in parametric QA, answers come from parameterized knowledge, and pathway fragility does not equal uncertainty. The paper gives quantitative envelope conditions: coverage ≤ 0.15, subnetwork size S ≈ 40.
- "Inspect the foundation, not the sign": traditional UE reads the output distribution (the sign); ASMI probes structural robustness (the foundation). The post argues this distinction — surface vs. structural signals — generalizes to other complex systems.
- Training-free: no retraining or extra labels; costs ~40 forward passes per query — pricier than a single inference, cheaper than sampling-consistency (which generates full responses).
- Another instance of the "evaluation blind-spot law": a single aggregate metric (average accuracy/calibration) hides critical failure modes. The post draws parallels to Progressive Cramming (99% token accuracy masking 100% generation failure), the Token Budget paper (bimodal 96.5% vs 11.5% outcomes), and QuantiBias (quantization introducing bias invisible to standard safety checks).
- High-stakes deployments (medical, legal, financial QA) where the cost of errors exceeds the cost of ~40 forward passes.
- Confidence routing: decide which queries need retrieval augmentation, direct answers, or human review.
- Model selection: comparing models on the "confident but fragile" subset is more informative than average accuracy.
Why It Matters
Honest Limitations
1. Compute overhead: ~40 forward passes per query; a Top-K masking approximation exists but loses accuracy. 2. Attention-only: MLP layers and residual-stream components are untested; attention-path fragility may be just the tip of the iceberg. 3. Narrow envelope: effective only on grounded QA; not a universal uncertainty estimator. 4. Model scale: tested mainly on 1B–8B models; behavior at larger scale is unknown.
Practical Implications
For RAG systems and grounded QA:
One-line Summary
A model's confidence comes in two flavors: robust confidence that survives perturbation, and fragile confidence that shatters at a touch. ASMI is the first training-free method to distinguish the two at the attention-pathway level — it doesn't replace traditional confidence, it covers the half of the world traditional confidence cannot see.
> Paper: arXiv:2608.11138 > Authors: Minsoo Kim, Sungyoung Ji, Kisung Moon, Ilyong Yoon > Published: 2026-08-11