Beyond Confidence: A Multi-Dimensional Self-Assessment Framework for LLMs Based on Cognitive Appraisal Theory
> In May 2026, Bhattacharyya et al. drew on Cognitive Appraisal Theory to propose a multi-dimensional view of LLM self-assessment. The study shows that the widely used single confidence dimension is an inconsistent and overly optimistic predictor of correctness, while capability-related dimensions (especially effort and ability) consistently match or exceed confidence's predictive power across a large experiment covering 12 LLMs, 38 tasks, and 8 domains. Effort additionally yields calmer estimates and remains stable across model scales, and the most informative dimension varies systematically with task characteristics — effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate retrieval-oriented ones. These results suggest structured multi-dimensional self-assessment is a promising path to more reliable and safer LLM deployment.
---
1. Background: The Reliability Crisis in LLM Self-Assessment
1.1 From Probabilities to Verbalized Confidence
LLM reliability assessment has evolved through two stages:
| Stage | Method | Limitation | |:---|:---|:---| | Probabilistic estimation | Maximum of output probability distribution | May assign high probability even to wrong answers | | Verbalized confidence | Model self-reports a 1–10 score | Overly optimistic, inconsistent |
1.2 Systematic Biases of Confidence
Prior research shows verbalized confidence in LLMs suffers from:
- Overconfidence: high confidence even on wrong answers
- Inconsistency: different confidence for the same question across repeated queries
- Poor calibration: weak correlation between confidence and actual accuracy
- Familiarity: past experience with similar questions
- Source certainty: origin of certainty (derivation vs. memory)
- Time pressure: perceived time pressure
- Task-adaptive weighting
- Online-learned combination functions
- Fusion with external verifiers
- High effort + wrong → strong negative signal
- Low effort + correct → weak positive signal
- Forming "effort-aware" reward shaping
- Assess effort at each step
- Trigger backtracking or expanded search when effort is anomalous
- Forming "metacognitive" inference control
---
2. The Multi-Dimensional Framework of Cognitive Appraisal Theory
2.1 Theoretical Origin
Cognitive Appraisal Theory is an established psychological framework holding that human self-evaluation is multi-dimensional:
| Dimension | Definition | Focus | |:---|:---|:---| | Effort | Cognitive resources invested | Process load | | Ability | Subjective assessment of task difficulty | Ability-task match | | Control | Sense of controllability over outcome | Agency | | Certainty | Belief in correctness (i.e., confidence) | Outcome belief | | Pleasantness | Subjective affective experience | Emotional response | | Attention | Concentration required | Cognitive demand |
2.2 Advantages of Multi-Dimensional Evaluation
Comparing single vs. multi-dimensional signals:
| Scenario | Confidence Signal | Effort + Ability Signal | Diagnostic Value | |:---|:---:|:---:|:---:| | Correct + high confidence | ✅ Consistent | Low effort + high ability | Same | | Correct + low confidence | ⚠️ Inconsistent | High effort + low ability | Higher | | Wrong + high confidence | ❌ Misleading | High effort + low ability | Higher | | Wrong + low confidence | ✅ Consistent | Low effort + low ability | Same |
> Key insight: contradictory multi-dimensional signals (e.g., high effort + high confidence) are themselves an error warning.
---
3. Large-Scale Empirical Study
3.1 Experimental Scale
| Dimension | Scale | |:---|:---:| | LLMs | 12 | | Tasks | 38 | | Domains | 8 | | Total evaluation points | ~3,600 |
3.2 Predictive Power Comparison
| Dimension | Mean AUROC | Over-optimism Bias | Cross-Scale Stability | |:---|:---:|:---:|:---:| | Confidence | 0.65 | High | Poor | | Effort | 0.72 | Low | High | | Ability | 0.70 | Medium | Medium | | Control | 0.62 | Medium | Medium | | Pleasantness | 0.55 | — | — | | Attention | 0.58 | — | — |
> Effort's overall advantage: highest predictive power + lowest over-optimism + best stability.
3.3 Task-Type Moderation Effects
The most informative dimension varies with task characteristics:
| Task Type | Example Tasks | Most Predictive Dimension | Explanation | |:---|:---|:---:|:---| | Reasoning-intensive | Math proofs, logical deduction | Effort | Process load directly reflects depth of understanding | | Retrieval-oriented | Knowledge QA, fact lookup | Ability / Confidence | Knowing vs. not knowing is binary | | Creative generation | Story writing, design | Effort + Pleasantness | Investment and felt experience both matter |
---
4. Why Is Effort More Reliable?
4.1 Differences in Signal Source
| Property | Confidence | Effort | |:---|:---:|:---:| | Based on | Surface plausibility of answer | Cognitive load of generation process | | When hallucinating | Possibly high (plausible surface) | Usually high (tried hard but still wrong) | | When guessing | Possibly high (blind confidence) | Usually low (little effort expended) | | Calibration | Poor | Good |
4.2 Cross-Scale Stability
Effort calibration remains stable across model scales:
| Model Scale | Confidence Calibration Error | Effort Calibration Error | |:---:|:---:|:---:| | 7B | 0.25 | 0.12 | | 13B | 0.22 | 0.11 | | 70B | 0.28 | 0.10 |
> Trend: larger models may have worse confidence (worsening overconfidence), while effort stays stable or even improves.
---
5. Relation to Related Work
5.1 Tracing Uncertainty (Round 17)
Round 17 predicted correctness via uncertainty profiles. Effort offers another "process signal" — reading quality information from subjective reports rather than statistical features.
5.2 POISE (Round 15)
POISE estimates value from internal states (hidden states + entropy statistics). Effort can be viewed as a "self-reported internal state" — the model's perception of its own cognitive load.
5.3 Rubric-Grounded RL (Round 19)
Rubric-Grounded RL showed multi-dimensional rewards beat binary rewards. This study shows multi-dimensional self-assessment likewise beats single confidence — both point to a "multi-dimensional > one-dimensional" principle.
5.4 VecCISC (Round 24)
VecCISC filters low-quality reasoning chains. Multi-dimensional self-assessment can serve as a filtering criterion: high effort + low confidence → trigger re-reasoning.
---
6. Limitations and Future Directions
6.1 Dimension Extensions
Are the current 6 dimensions sufficient? Potential additions:
6.2 Combination Models
Optimal combination of multi-dimensional signals:
Directions:
6.3 Training-Time Applications
Integrating effort signals into RLVR training:
6.4 Real-Time Inference Control
Dynamic self-monitoring during generation:
7. Conclusion
By introducing a multi-dimensional framework grounded in Cognitive Appraisal Theory, *Beyond Confidence* provides a new theoretical foundation and practical path for LLM self-assessment. Core contributions:
1. Problem diagnosis: systematic unreliability of single confidence 2. Theoretical framework: a 6-dimensional cognitive appraisal model 3. Empirical scale: 12 models × 38 tasks × 8 domains 4. Key finding: effort's overall advantage (predictive power + calibration + stability) 5. Contextual adaptation: the best dimension varies with task type
As LLM deployment increasingly involves high-stakes decisions, reliable self-assessment is no longer a bonus but a necessity. Multi-dimensional self-assessment offers an actionable path toward more reliable and safer human-machine collaboration systems.
---
Paper Details
| Item | Content | |:---|:---| | Title | Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs | | Authors | Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, James Z. Wang | | Affiliation | Pennsylvania State University et al. | | arXiv ID | 2605.07806 | | Date | 2026-05-08 | | Core Contribution | Applying cognitive appraisal theory to LLM self-assessment; 6-dimensional framework; 12 LLMs × 38 tasks × 8 domains; effort/ability outperform confidence; task-type moderation effects | | Key Results | Effort AUROC 0.72 vs. confidence 0.65; effort shows lowest over-optimism; stable across scales; effort most predictive for reasoning-intensive tasks |
#Research #SelfAssessment #Calibration #CognitiveAppraisal #MultiDimensional #Reliability