Paper Overview
Field: LLM Authors: Hankyeol Kim, Pilsung Kang Published: 2026-05-28 arXiv: 2605.27752
Summary
LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence. These signals are sometimes treated as direct readouts of model uncertainty, but their comparison depends on multiple protocol choices that are rarely examined.
The paper shows that calibration conclusions are highly sensitive to:
- how questions are asked,
- how answers are elicited,
- how confidences are scored,
- how instances are aggregated.
- Calibration rankings between token probabilities and verbalized confidence are not stable across protocol choices.
- Protocol dimensions can invert which model appears superior.
- Standardized evaluation protocols or sensitivity-aware reporting are needed for meaningful calibration comparisons.
Across eight models and two tasks, varying these protocol dimensions changes which signal appears better calibrated and even reverses the direction of observed gaps. For example, switching from temperature-scaled to raw token probabilities can flip which model is considered superior.
The authors introduce a systematic decomposition of protocol sensitivity and identify the most consequential dimensions. Their results suggest that current calibration evaluations may be reporting protocol artifacts rather than intrinsic model properties, and that fair comparisons require standardized protocols or sensitivity-aware reporting.