Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence and RegimeAbstain

Forum topic · 小凯 · 2026-09-22

Summary

Multi-hop retrieval failures are not uniformly distributed across queries but cluster in structurally predictable subpopulations. This arXiv paper (2609.22056) by Andre Bacellar formalizes this structure with two results: CWAR Reducibility, showing confident-failure reduction is achievable if and only if retrieval features carry mutual information about success (satisfied by LLM-judge pipelines but weaker in dense-only settings); and Feature Regime Complementarity, showing no single ANN score feature is best across all failure regimes. The authors instantiate these principles as RegimeAbstain, which computes a Retrieval Confidence Score (RCS) from up to nine query-ANN structural features with no extra LLM calls, and implements a calibrated abstention policy. Evaluated on MuSiQue, 2WikiMultiHopQA, and HoVer across five failure regimes (CWAR 14.5%–62.1%), RCS achieves best-or-tied AUC-AC versus eight confidence baselines, cutting CWAR from 39.5% to 20.6% at 50% coverage on MuSiQue (LLM-judge), with cross-domain transfer losing only 0.5 AUC points.

Overview

Field: NLP Author: Andre Bacellar Published: 2026-09-18 arXiv: 2609.22056

Abstract (translated)

Multi-hop retrieval failures are not uniformly distributed across queries: they cluster in structurally predictable subpopulations. The paper proves two results formalizing this structure.

1. CWAR Reducibility: Confident-failure reduction is achievable if and only if retrieval features carry mutual information about success. LLM-judge pipelines satisfy this condition, but it is substantially weaker in dense-only settings, explaining the AUC-AC gap between the two regimes.

2. Feature Regime Complementarity: No single ANN score feature achieves best predictive performance across all failure regimes. The dominant feature differs between datasets (query length on MuSiQue, hop-1 concentration on HoVer), and a constructive witness pair shows each feature is necessary in one regime and non-contributory in the other.

These principles are instantiated as RegimeAbstain, which computes a Retrieval Confidence Score (RCS) — a logistic function over up to nine query-ANN structural features, all obtainable without any additional LLM calls — and uses it for a calibrated abstention policy.

Evaluation

The paper defines the Confident Wrong Answer Rate (CWAR) metric and evaluates on three multi-hop benchmarks (MuSiQue, 2WikiMultiHopQA, HoVer) and two retrieval architectures (LLM-judge and dense-only), covering five failure regimes with CWAR from 14.5% to 62.1%.

Key results:

RCS achieves best-or-tied AUC-AC across all five conditions, compared against eight confidence baselines.
On MuSiQue (LLM-judge), RCS reduces CWAR from 39.5% to 20.6% at 50% coverage — a 47.8% relative reduction — with ECE = 0.035.
A model trained on MuSiQue transfers to 2WikiMultiHopQA with only 0.5 AUC points lost, confirming that regime features have cross-domain structure.

Original Abstract

Multi-hop retrieval failures are not uniformly distributed across queries: they cluster in structurally predictable subpopulations. We prove two results formalizing this structure. First (CWAR Reducibility): confident-failure reduction is achievable if and only if retrieval features carry mutual information about success, a condition satisfied by LLM-judge pipelines but substantially weaker in dense-only settings, explaining the AUC-AC gap between regimes. Second (Feature Regime Complementarity): no single ANN score feature achieves best predictive performance across all failure regimes; the dominant feature differs between datasets (query length on MuSiQue, hop-1 concentration on HoVer), and a constructive witness pair shows each is necessary in one regime and non-contributory in the other.

--- *Auto-collected on 2026-09-22*