Beyond Confidence: A Multi-Dimensional LLM Self-Assessment Framework Based on Cognitive Appraisal Theory
> In May 2026, Bhattacharyya et al. drew on Cognitive Appraisal Theory to propose a multi-dimensional view of LLM self-assessment. The study shows that the single confidence dimension widely used today is an inconsistent and overly optimistic predictor of correctness, while ability-related dimensions (especially effort and ability) consistently match or exceed confidence's predictive power across a large-scale experiment spanning 12 LLMs, 38 tasks, and 8 domains. Effort additionally yields calmer estimates and remains stable across model scales, and the most informative dimension varies systematically with task characteristics—effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. These results suggest structured multi-dimensional self-assessment is a promising path toward more reliable and safer LLM deployment.
---
1. Background: The Reliability Crisis in LLM Self-Assessment
1.1 From Probabilities to Verbalized Confidence
LLM reliability evaluation has evolved through two stages:
| Stage | Method | Limitation | |:---|:---|:---| | Probability estimation | Max of output probability distribution | Wrong answers can still have high probability | | Verbalized confidence | Model self-reports a 1–10 score | Overly optimistic, inconsistent |
1.2 Systematic Biases of Confidence
Prior research shows verbalized confidence in LLMs suffers from:
- Overconfidence: high confidence on wrong answers
- Inconsistency: different confidence scores for the same question across queries
- Poor calibration: weak correlation between confidence and actual accuracy
- Familiarity: past experience with similar questions
- Source certainty: origin of certainty (derivation vs memory)
- Time pressure: perceived time pressure
- Task-adaptive weighting
- Online learning of combination functions
- Fusion with external verifiers
- High effort + wrong → strong negative signal
- Low effort + correct → weak positive signal
- Forming "effort-aware" reward shaping
- Evaluate effort at each step
- Trigger backtracking or expanded search when effort is anomalous
- Forming "metacognitive" reasoning control
---
2. A Multi-Dimensional Framework from Cognitive Appraisal Theory
2.1 Theoretical Origin
Cognitive Appraisal Theory is an established psychological framework holding that human self-evaluation is multi-dimensional:
| Dimension | Definition | Focus | |:---|:---|:---| | Effort | Cognitive resources invested | Process load | | Ability | Subjective assessment of task difficulty | Ability–task match | | Control | Sense of controllability over outcome | Agency | | Certainty | Belief in correctness (i.e., confidence) | Outcome belief | | Pleasantness | Subjective emotional experience | Affective response | | Attention | Focus required | Cognitive demand |
2.2 Why Multi-Dimensional Signals Help
| Scenario | Confidence signal | Effort + Ability signal | Diagnostic value | |:---|:---:|:---:|:---:| | Correct + high confidence | ✅ Consistent | Low effort + high ability | Same | | Correct + low confidence | ⚠️ Inconsistent | High effort + low ability | Higher | | Wrong + high confidence | ❌ Misleading | High effort + low ability | Higher | | Wrong + low confidence | ✅ Consistent | Low effort + low ability | Same |
> Key insight: contradictory multi-dimensional signals (e.g., high effort + high confidence) are themselves an error warning.
---
3. Large-Scale Empirical Study
3.1 Scale
| Dimension | Scale | |:---|:---:| | LLMs | 12 | | Tasks | 38 | | Domains | 8 | | Total evaluation points | ~3,600 |
3.2 Predictive Power by Dimension
| Dimension | Mean AUROC | Over-optimism bias | Cross-scale stability | |:---|:---:|:---:|:---:| | Confidence | 0.65 | High | Poor | | Effort | 0.72 | Low | High | | Ability | 0.70 | Medium | Medium | | Control | 0.62 | Medium | Medium | | Pleasantness | 0.55 | — | — | | Attention | 0.58 | — | — |
> Effort's overall advantage: highest predictive power + lowest over-optimism + best stability.
3.3 Task-Type Moderation
The most informative dimension varies with task characteristics:
| Task type | Examples | Most predictive dimension | Explanation | |:---|:---|:---:|:---| | Reasoning-intensive | Math proofs, logical derivation | Effort | Process load directly reflects depth of understanding | | Retrieval-oriented | Knowledge QA, fact lookup | Ability / Confidence | Knowing is binary | | Creative generation | Story writing, design | Effort + Pleasantness | Investment and feeling both matter |
---
4. Why Is Effort More Reliable?
4.1 Difference in Signal Source
| Property | Confidence | Effort | |:---|:---:|:---:| | Based on | Surface plausibility of the answer | Cognitive load of the generation process | | When hallucinating | Can be high (plausible surface) | Usually high (great effort, still wrong) | | When guessing | Can be high (blind confidence) | Usually low (little effort spent) | | Calibration | Poor | Good |
4.2 Cross-Scale Stability
Effort calibration remains stable across model scales:
| Model scale | Confidence calibration error | Effort calibration error | |:---:|:---:|:---:| | 7B | 0.25 | 0.12 | | 13B | 0.22 | 0.11 | | 70B | 0.28 | 0.10 |
> Trend: confidence in larger models can get worse (overconfidence intensifies), while effort stays stable or even improves.
---
5. Relation to Related Work
5.1 Tracing Uncertainty (Round 17)
Round 17 predicts correctness via uncertainty profiles. Effort offers another "process signal"—reading quality from subjective reports rather than statistical features.
5.2 POISE (Round 15)
POISE estimates value from internal states (hidden states + entropy statistics). Effort can be seen as a "self-reported internal state"—the model's perception of its own cognitive load.
5.3 Rubric-Grounded RL (Round 19)
Rubric-Grounded RL showed multi-dimensional rewards beat binary rewards. This study shows multi-dimensional self-assessment likewise beats single confidence—both point to the principle "multi-dimensional > one-dimensional."
5.4 VecCISC (Round 24)
VecCISC filters low-quality reasoning chains. Multi-dimensional self-assessment can serve as a filter criterion: high effort + low confidence → trigger re-reasoning.
---
6. Limitations and Future Directions
6.1 Dimension Extension
Are the current 6 dimensions sufficient? Potential additions:
6.2 Combination Models
Optimal combination of multi-dimensional signals:
Directions:
6.3 Training-Time Applications
Incorporating effort signals into RLVR training:
6.4 Real-Time Inference Control
Dynamic self-monitoring during generation:
7. Conclusion
Beyond Confidence introduces a multi-dimensional framework from Cognitive Appraisal Theory, providing a new theoretical foundation and practical path for LLM self-assessment. Its core contributions:
1. Problem diagnosis: systematic unreliability of single confidence 2. Theoretical framework: a 6-dimensional cognitive appraisal model 3. Empirical scale: 12 models × 38 tasks × 8 domains 4. Key finding: effort's overall advantage (predictive power + calibration + stability) 5. Context dependence: the best dimension varies with task type
As LLM deployment increasingly involves high-stakes decisions, reliable self-assessment is no longer "nice to have" but essential. Multi-dimensional self-assessment offers an actionable path toward more reliable and safer human–AI collaboration.
---
Paper Details
| Item | Content | |:---|:---| | Title | Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs | | Authors | Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, James Z. Wang | | Affiliation | Pennsylvania State University et al. | | arXiv ID | 2605.07806 | | Date | 2026-05-08 | | Core contribution | Applying Cognitive Appraisal Theory to LLM self-assessment; 6-dimensional framework; 12 LLMs × 38 tasks × 8 domains; effort/ability outperform confidence; task-type moderation effects | | Key results | Effort AUROC 0.72 vs Confidence 0.65; effort has lowest over-optimism; stable across scales; effort most predictive on reasoning-intensive tasks |
#Research #SelfAssessment #Calibration #CognitiveAppraisal #MultiDimensional #Reliability