Summary
A new arXiv paper (2605.27752) by Hankyeol Kim and Pilsung Kang examines how evaluation protocol choices affect LLM confidence calibration results. Calibration is typically assessed by comparing token-probability scores with verbalized confidence, but the authors show these conclusions are highly sensitive to how questions are asked, how answers are elicited, how confidence is scored, and how instances are aggregated. Across eight models and two tasks, varying these protocol dimensions changes which signal appears better calibrated and can even reverse the direction of observed gaps. Notably, switching from temperature-scaled to raw token probabilities can flip which model is considered superior. The paper introduces a systematic decomposition of protocol sensitivity and identifies the most consequential dimensions, suggesting that current calibration evaluations may report protocol artifacts rather than intrinsic model properties, and that fair comparison requires standardized protocols or sensitivity-aware reporting.
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
Authors: Hankyeol Kim, Pilsung Kang
arXiv: 2605.27752
Key points
- LLM confidence calibration is commonly evaluated by comparing two signals: token-probability scores and verbalized confidence. These are sometimes treated as direct readouts of model uncertainty, but their comparison depends on protocol choices that are rarely examined.
- Calibration conclusions are highly sensitive to:
- how questions are asked,
- how answers are elicited,
- how confidence is scored,
- how instances are aggregated.
- Across eight models and two tasks, varying these protocol dimensions changes which signal appears better calibrated and can even reverse the direction of observed gaps.
- Example: switching from temperature-scaled to raw token probabilities can flip which model is considered superior.
- The paper introduces a systematic decomposition of protocol sensitivity and identifies the most consequential dimensions.
Implications
Current calibration evaluations may be reporting protocol artifacts rather than intrinsic model properties. Fair comparisons between signals and models require standardized protocols or sensitivity-aware reporting.
*Collected automatically on 2026-05-29.*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177980522