LLM Confidence Is Overrated: Effort Predicts Errors Better Across 12 Models and 38 Tasks

> Core claim: Bhattacharyya et al. (2026) apply psychology's Cognitive Appraisal Theory to LLM self-assessment. Asking a model "how confident are you?" yields inconsistent, overly optimistic answers—models often rate wrong answers 9/10. But asking "how much effort did this take?" (effort) or "how hard was this for you?" (ability) predicts errors far better. Across 12 LLMs, 38 tasks, and 8 domains, effort and ability consistently matched or exceeded confidence as error predictors, and effort estimates were calmer and less inflated. If correct, every system relying on model self-reported confidence—including RLVR reward design and human-AI collaboration interfaces—may be using the wrong signal.

1. Why Confidence Fails

The current mainstream approach: ask the model to rate its answer confidence on a 1-10 scale.

| Scenario | Model's answer | Actual correctness | Problem | |:---|:---:|:---:|:---| | Easy question | "10/10 confident" | ✅ Correct | Fine | | Hard question (correct) | "7/10" | ✅ Correct | Overly conservative | | Hard question (wrong) | "9/10" | ❌ Wrong | Overly optimistic | | Hallucination | "8/10" | ❌ Wrong | Severely overconfident |

The core problem: Confidence is not a reliable calibration signal. Models frequently assign high confidence to wrong answers—the well-known overconfidence problem.

Suggested causes:

| Cause | Explanation | |:---|:---| | Training bias | SFT teaches models to "state things confidently" even when unsure | | Lack of metacognition | Models lack a genuine sense of "knowing what they know" | | Pattern matching | Confidence reflects surface features (e.g., answer format), not deep understanding |

2. Cognitive Appraisal Theory: A Lens Borrowed from Psychology

Cognitive Appraisal Theory holds that human self-assessment is multi-dimensional, not single-dimensional:

| Dimension | Meaning | Example question | |:---|:---|:---| | Effort | Cognitive resources invested | "How much work was this?" | | Ability | Subjective difficulty assessment | "How hard was this for me?" | | Control | Sense of control over the outcome | "Can I control answer quality?" | | Certainty | Belief in answer correctness | "How sure am I?" (i.e., confidence) | | Pleasantness | Subjective experience of solving | "Was this pleasant?" | | Attention | Focus required | "Did this need full concentration?" |

Why is multi-dimensional better? Single-dimension confidence is like a dashboard with one needle—it can get stuck on "high." Multi-dimensional assessment is like six needles: when effort is high but confidence is also high, that contradiction itself is a warning signal.

3. Large-Scale Experiment: 12 Models × 38 Tasks × 8 Domains

Scale

| Dimension | Scale | |:---|:---:| | LLMs | 12 | | Tasks | 38 | | Domains | 8 |

Key findings

| Dimension | Error-prediction power | Over-optimism | Stability across scales | |:---|:---:|:---:|:---:| | Confidence | Moderate | High | Unstable | | Effort | High | Low | Stable | | Ability | High | Moderate | Fairly stable | | Control | Moderate | Moderate | Average | | Pleasantness | Low | — | — | | Attention | Low | — | — |

> Effort is the champion: in most settings, effort matches or exceeds confidence in predictive power while being calmer and more stable.

Task-type moderation

The most informative dimension varies with task characteristics:

| Task type | Most predictive dimension | Explanation | |:---|:---:|:---| | Reasoning-intensive (math, logic) | Effort | High effort + wrong = genuinely incapable | | Retrieval-oriented (knowledge QA) | Ability / Confidence | You either know it or you don't | | Creative generation | Effort + Pleasantness | Both investment and experience matter |

> Key insight: there is no universal self-assessment dimension—the best one depends on task type.

4. Why Effort Is More Reliable Than Confidence

| Property | Confidence | Effort | |:---|:---:|:---:| | Based on | Surface plausibility of the answer | Cognitive load of the solution process | | When hallucinating | Possibly high (answer "looks right") | Usually high (high effort but still wrong) | | When guessing | Possibly high (blind confidence) | Usually low (little effort spent) | | Calibration | Poor | Good |

> Intuition: Effort is a *process signal*—it reflects how much the model struggled. Struggled and got it right = capability sufficient; struggled and got it wrong = capability insufficient. This is more reliable than "I feel like I'm right."

Effort calibration is also stable across model scales—from 7B to 70B, "how much effort" remains a reliable signal, while confidence calibration may even worsen with scale (worsening overconfidence).

5. Connections to Related Work

• Tracing Uncertainty (Round 17): uses uncertainty profiles to predict answer correctness; effort offers another "process signal" read from generation dynamics.
• POISE (Round 15): uses internal states for value estimation; effort can be seen as a self-report of perceived internal cognitive load.
• Rubric-Grounded RL (Round 19): replaces binary rewards with multi-dimensional rubrics; this study similarly shows multi-dimensional > single-dimensional self-assessment.
• VecCISC (Round 24): filters low-quality reasoning chains; effort can serve as a filter criterion—high effort + low confidence = possibly needs re-reasoning.

6. The Author's Bet

The author bets $1,000: by the end of 2026, multi-dimensional self-assessment (especially effort + ability + confidence combined) will replace single confidence as the standard for LLM reliability evaluation.

Why:

1. Large experimental scale: 12 models × 38 tasks × 8 domains ≈ 4,000 model-task combinations—not a small sample. 2. Solid theoretical foundation: Cognitive Appraisal Theory is a mature 40-year-old psychological framework. 3. Simple to implement: just add a few prompt questions—no new model training needed. 4. Compatible with existing systems: can augment any system that needs self-assessment. 5. Practical stakes: high-risk scenarios like human-AI collaboration, medical diagnosis, and legal advice need reliable self-assessment.

7. Limitations and Future Directions

• Dimension selection: Are there more valuable dimensions beyond the six (e.g., "familiarity," "source of certainty"—derived vs. memorized)?
• Combination strategies: simple average, weighted average, task-type-dependent dynamic weights, or learned combiners?
• Integration with RL: Could RLVR use effort signals? High effort + wrong = large negative reward; low effort + correct = small positive reward.
• Real-time calibration: monitoring effort during generation to dynamically adjust length or trigger backtracking.

Regardless, the paper proposes a simple but profound shift: don't just ask the model "how certain are you"—also ask "how much effort did this take." The latter is often more honest.

Paper Details

| Item | Detail | |:---|:---| | Title | Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs | | Authors | Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, James Z. Wang | | Affiliation | Pennsylvania State University et al. | | arXiv ID | 2605.07806 | | Date | 2026-05-08 | | Core contribution | Applies Cognitive Appraisal Theory to LLM self-assessment; 6-dimension framework; 12 LLMs × 38 tasks × 8 domains; effort/ability outperform confidence; task-type moderation effects | | Key results | Effort and ability consistently match/exceed confidence; effort estimates are calmer and stable across scales; effort most predictive on reasoning-intensive tasks; ability/confidence dominate retrieval-oriented tasks |