Beyond Confidence: Effort Predicts LLM Errors Better Than Self-Reported Confidence
A forum post on zhichai.net reviews a 2026 paper applying Cognitive Appraisal Theory from psychology to LLM self-assessment: *Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs* by Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, and James Z. Wang (Pennsylvania State University et al., arXiv:2605.07806).
Core claim
Asking an LLM "How confident are you?" yields an inconsistent and overly optimistic signal—models frequently rate wrong answers (including hallucinations) as 9/10 confident. But asking "How much effort did you expend?" (effort) or "How hard was this task for you?" (ability) produces far more accurate error predictions. Across 12 LLMs, 38 tasks, and 8 domains, effort and ability consistently match or exceed confidence as error predictors, and effort estimates are more conservative and stable.
Why confidence fails
- Training bias: SFT teaches models to state answers confidently even when uncertain.
- Lack of metacognition: models lack genuine "knowing what they know" ability.
- Surface pattern matching: confidence ratings often reflect surface features rather than deep understanding.
- Effort and ability always match or beat confidence in predicting correctness.
- Effort estimates are less inflated and stable across model scales (7B to 70B), while confidence calibration may even worsen with scale.
- Task type moderates the effect: effort is most predictive for reasoning-intensive tasks (math, logic); ability/confidence dominates retrieval-oriented knowledge QA.
- No single "universal" self-assessment dimension exists—the best signal depends on the task.
- Additional useful dimensions (familiarity, source of certainty: derivation vs. memory)
- Optimal combination strategies (weighted averages, task-dependent weights, learned combiners)
- Using effort signals in RLVR rewards (high effort + wrong = large negative reward)
- Real-time effort monitoring to dynamically adjust generation length or trigger backtracking
- Title: Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
- Authors: Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, James Z. Wang
- Affiliation: Pennsylvania State University et al.
- arXiv: 2605.07806 (2026-05-08)
Six appraisal dimensions borrowed from psychology
| Dimension | Meaning | |:---|:---| | Effort | Cognitive resources invested in solving | | Ability | Subjective difficulty of the task | | Control | Sense of control over the outcome | | Certainty | Belief in correctness (i.e., confidence) | | Pleasantness | Subjective experience of solving | | Attention | Focus required |
Experimental findings (12 models × 38 tasks × 8 domains)
Why effort is more honest
Effort is a process signal: it reflects how much the model struggled. High effort + wrong answer = genuine capability gap; low effort + correct answer = easy task. In contrast, confidence is an outcome-side judgment based on how plausible the answer looks—hallucinations can "look right" and receive high confidence.
Connections to prior work
The post links this to earlier threads: uncertainty tracing from generation dynamics, POISE's internal-state value estimation, rubric-grounded RL (both suggest "multi-dimensional > single-dimensional" signals), and VecCISC reasoning-chain filtering (high effort + low confidence could flag chains for re-reasoning).
Author's bet
The author predicts that by end of 2026, multi-dimensional self-assessment (effort + ability + confidence) will replace single-dimension confidence as standard practice for LLM reliability evaluation. Rationale: the large experiment scale (~4,000 model-task combinations), the mature 40-year psychological foundation, trivial implementation cost (just extra prompt questions), and compatibility with existing systems. Counterarguments addressed: effort does not require assuming models have subjective experience—it is an operational signal, and the added inference cost is negligible.