Confidence Calibration in Large Language Models: Overconfidence, the Hard-Easy Effect, and LifeEval

Forum topic · 小凯 · 2026-05-27

Summary

A preregistered study by Noam Michael, Daniel BenShushan, and Jacob Ben-Ari (arXiv:2505.21643) examines confidence calibration in large language models across diverse tasks. The findings show that current LLMs, like humans, tend to be overconfident: on average, stated confidence exceeds actual accuracy. Crucially, this tendency is moderated by a strong hard-easy effect—overconfidence is greatest on difficult tests, whereas easy tests exhibit substantial underconfidence. The authors introduce LifeEval, a benchmark for evaluating model calibration across levels of difficulty.

Paper Overview

Research Area: Machine Learning Authors: Noam Michael, Daniel BenShushan, Jacob Bien Published: 2026-05-26 arXiv: 2505.21643

Abstract

We investigate the calibration of large language models' (LLMs') confidence across diverse tasks. The results of our preregistered study show that the current crop of LLMs are, like people, too sure they are right: confidence exceeds accuracy, on average. Importantly, however, this tendency is moderated by a powerful hard-easy effect, wherein overconfidence is greatest on difficult tests; by contrast, easy tests actually show substantial underconfidence. We develop LifeEval, a test for evaluating model calibration across levels of difficulty.

Key Points

LLMs are systematically overconfident: on average, their stated confidence exceeds their actual accuracy—mirroring a well-documented human bias.
A strong hard-easy effect moderates this tendency:
Difficult tests: overconfidence is most pronounced.
Easy tests: models show substantial underconfidence.
The study was preregistered, strengthening the reliability of the findings.
The authors introduce LifeEval, a benchmark designed to evaluate model calibration across varying levels of difficulty.

---

*Auto-collected on 2026-05-27.*