RADAR: Repair, Adopt, Diagnose — Why a Perfectly Calibrated LLM Confidence Score Can Still Be Systematically Wrong

Forum topic · 小凯 · 2026-10-18

Summary

This Chinese forum post analyzes the RADAR framework (arXiv:2610.22139) for post-hoc confidence repair in LLMs. It opens with a medical analogy: a thermometer can pass quality inspection (perfect calibration) yet consistently misread a specific 20% subgroup — mirroring how Expected Calibration Error (ECE) can be zero while errors remain predictable by input family. The post decomposes squared-loss risk of any scalar repaired score into outcome noise (N), grouping loss (I), and mapping error, plus an order-preserving pooling gap (G), showing that well-calibrated repairs still lose I+G in ways calibration checks cannot see. It details PAIC (Probe-Anchored Isotonic Calibration), which distills a rich, trained correctness probe onto the scalar score's scale using unlabeled calibration data, with near minimax-optimal isotonic rates. A key theorem: what matters is not a probe's overall accuracy but whether its errors average out within each base-score level — probes with equal MSE produced excess repair risks ranging from 3.9×10⁻⁷ to 0.0064. Adoption decisions use betting-based sequential tests (vs. Bernstein bounds: 42.95% vs 15.09% correct adoptions over 47,400 replays), and audits use cluster-adjusted kernel calibration error (SKCE) with wild bootstrap to hunt systematic residuals by feature family. Experiments on Llama 3.2, Qwen 2.5, and Mistral 7B show rich predictors serving directly beat scalar projections, but distilling back to a scalar under deployment constraints erases the gains, and lower Brier risk does not guarantee better review decisions.

RADAR: Repairing and Auditing LLM Confidence Scores — Why Calibration Alone Is Not Enough

This post is a detailed walkthrough of the RADAR paper (arXiv:2610.22139), framed as part two of a "verifier economics" series. Part one diagnosed evaluation guards; this part covers repairing confidence scores — and argues that a repaired score must be verified and audited before it can be trusted again.

Key points

A perfectly calibrated score can be systematically wrong. The post opens with a thermometer that passes QC — every confidence bucket's average confidence exactly equals its accuracy, so ECE = 0 — yet consistently misreads 20% of patients with unusual "fever patterns." Similarly, an LLM can report confidence 0.8 with exactly 80% accuracy while a whole error family (e.g., converting minutes to hours) is invisibly wrong.
Risk decomposition. For any scalar repaired score under squared loss:

> L(T) = N + I + G

where N is outcome noise (irreducible), I is the grouping loss — information about correctness hidden by the scalar score itself — and G is the pooling gap imposed by order-preserving repairs. Only mapping error is fixable by standard recalibration (temperature scaling, Platt scaling, isotonic regression). The optimal order-preserving repair is well calibrated (ECE = 0) yet always loses I + G — a loss invisible to any method that checks its own calibration.

PAIC (Probe-Anchored Isotonic Calibration). Train a rich correctness probe g_n on labeled training data; then fit a monotone map τ on n *unlabeled* calibration prompts (drawn from the target population) that distills the probe's predictions back onto the original score scale:

> minimize Σ_j Σ_i (τ(P_ji) − g_n(Z_ji))², subject to τ monotone

At serving time only τ̂_n(P) is computed — downstream thresholds, queues, and APIs are untouched. Theorem 3.2 gives convergence to the optimal monotone repair once the probe's *score-conditional bias* vanishes, with excess error E[u_n] ≤ 25/√n, sharpening to the minimax-optimal O(n^(−2/3)) rate for isotonic regression.

Where a probe errs matters more than how accurate it is. Three probes with identical excess outcome MSE (0.0064) produced excess repair risks of 3.90×10⁻⁷, 3.16×10⁻⁵, and 0.0064 after PAIC fitting — errors that cancel within each score level are harmless; constant per-level offsets are fatal. Even a "dumb" probe predicting 0.8 for everything yields the optimal scalar repair target, because its conditional bias per score level is zero.
Adoption via betting, not blind replacement. Prediction-powered inference supplies conditional-unbiased risk-difference estimates from an unlabeled pool plus a small labeled correction pool. A betting-based e-process certifies a candidate only when

> E_h = (1/|S|) Σ_s Π_j (1 − s·D_jh / B_h) ≥ H/α

guaranteeing at most α false-adoption probability. Across 47,400 replays of the original score assignment, paired betting adopted a genuinely better candidate 42.95% of the time vs. 15.09% for one-sided empirical Bernstein intervals — conservatism and recklessness are two sides of the same coin.

Auditing by error family. Repaired scores can still hide residuals like −0.8 inside the unit-conversion family. RADAR audits with kernel calibration error:

> SKCE_k(s) = ‖E[(Y−s(Z))Φ(Z)]‖²

The kernel defines the "search warrant" — text-similarity kernels catch unit-conversion errors; answer-length kernels never will. Because residuals within one prompt are correlated (clustered sampling), the statistic averages within-prompt features and multiplies only across J independent prompts:

> SKCÊ_{k,c} = (1/J(J−1)) Σ_{j≠ℓ} ⟨ζ_j, ζ_ℓ⟩

Theorem 3.4: unbiased under arbitrary within-prompt dependence, with wild-bootstrap inference attaining asymptotic size α.

Three counterintuitive experimental findings

1. Perfect calibration ≠ trustworthy. On Llama 3.2 3B, Qwen 2.5 3B/7B, and Mistral 7B (3,270 BoolQ + 1,172 ARC Challenge prompts, 13 predictors), serving rich predictors directly beat monotone projection to a scalar by 0.0121–0.0169 Brier risk — but once distilled back to a scalar (PAIC's deployment constraint), no joint interval could prove superiority over the pure scalar baseline. 2. Transfer gains are fragile. Tree-based PAIC cut mean squared risk 4.9–16.4% vs. source-label isotonic across six shift settings; refitting on 777 unlabeled level-3 MATH prompts helped Llama overall (+0.00633 vs. source PAIC) but *hurt* on Prealgebra (+0.00456), and was inconclusive on Qwen. 3. Lower prediction loss ≠ better decisions. An appendix review/cost analysis shows improved Brier risk does not necessarily improve triage decisions — thresholds, cost matrices, and human review all couple in.

Deployment takeaways

RADAR = Repair (PAIC), Adopt (betting), Diagnose (kernel audit):

Repair only fixes what the scalar can express; the grouping loss I is a structural, unfixable deficit.
Betting keeps false adoption ≤ α but is conservative — labeled-pool budget must be spent carefully.
Audits only cover directions spanned by the kernel features; errors outside that space hide legally.

The closing message: "repair when you can, admit when you can't, and say where the mines might still be buried" is far more honest than "ECE = 0, ship it." A teaser suggests a follow-up called PROBE, turning periodic audits into an active, adversarial resident inspector.

References cited

arXiv:2610.22139 (RADAR); Guo et al. 2017 (calibration/ECE); Zadrozny & Elkan 2002 (isotonic); Perez-Lebel et al. 2023 (grouping loss); Angelopoulos et al. 2023 (prediction-powered inference); Waudby-Smith & Ramdas 2024 (betting); Widmann et al. 2019 and Marx et al. 2023 (kernel calibration tests); Zhang 2002 and Chatterjee et al. 2015 (isotonic risk bounds); Hinton et al. 2015 (distillation).