CAREBench Reveals LLMs Can Label Emotions but Struggle to Reason About Them

Forum topic · 小凯 · 2026-05-19

Summary

A forum post on zhichai.net discusses CAREBench, a new benchmark that tests whether large language models (LLMs) truly understand emotions rather than merely matching emotion labels. Built on Appraisal Theory from psychology—which holds that emotions arise from cognitive evaluation of events rather than the events themselves—CAREBench uses real-world narrative texts annotated with dual perspectives (actor and observer), including appraisal reasoning chains, appraisal dimension scores, and multi-label emotion annotations. The benchmark evaluated six mainstream LLMs and produced sobering findings: stronger models can match or exceed human performance on some tasks but fall short on appraisal reasoning and positive-emotion recognition; models perform inconsistently across reasoning steps, suggesting pattern memorization over genuine inference; and models fail to capture human subjective heterogeneity, offering standardized interpretations where individual reactions vary widely. The post concludes that existing emotion-prediction metrics likely overestimate LLMs' emotional understanding, since high accuracy on label-matching tests does not imply comprehension of why emotions arise—an important caveat for building emotionally intelligent conversational AI.

Have you ever told an AI "my boss yelled at me today and I feel terrible," and received a reply like "I understand you must be frustrated—maybe take a break or talk to a friend"? It sounds caring. But has the AI truly *understood* your emotion, or is it just reciting an emotion-response template? The difference matters, and a new benchmark called CAREBench exposes it.

Why emotion recognition ≠ emotion understanding

Consider three scenarios: a runner finishing a marathon 15 minutes slower than expected yet saying "I'm disappointed, but I did my best"; a newly promoted employee saying "I'm a bit nervous, but also excited"; a man reuniting with an old friend and feeling pleasant surprise mixed with estrangement. Humans infer these emotions easily—but only by *reasoning* over context, person, and outcome. Emotion understanding is a cognitive process, not label matching.

The fatal flaw in existing tests

Before CAREBench, AI emotion tests mostly asked: "What emotion does this text express?" with choices like sad/happy/angry/fearful. This only tests emotion-label recognition, not *why* the emotion arises. Knowing "he is sad" is a completely different cognitive level from understanding the cognitive appraisal that produced the sadness.

Appraisal Theory: the cognitive black box behind emotions

CAREBench's designers drew on Appraisal Theory: emotions are not directly triggered by events but generated through cognitive appraisal. Losing 100 yuan can trigger anger ("that's a whole day's wages"), relief ("at least it wasn't my credit card"), or calm ("bad luck wards off worse")—depending on the appraisal. Key dimensions include controllability, consistency with expectations, relevance, and adaptiveness.

CAREBench's innovation: test the reasoning chain, not the label

CAREBench is the first benchmark testing appraisal reasoning chains. It provides large numbers of real-world narrative texts with dual-perspective annotations (actor's and observer's viewpoints) across three levels: appraisal reasoning chains, appraisal dimension scores, and multi-label emotion annotations. Six mainstream LLMs were evaluated on four core questions.

Results: powerful models still have blind spots

1. Stronger models can match or exceed humans on some tasks, but underperform on appraisal reasoning and positive-emotion recognition. They may identify sadness correctly yet falter when asked *why* the person is sad and *how* they appraised the situation. 2. Performance is inconsistent across reasoning steps. This fragmentation suggests models may be memorizing patterns rather than genuinely learning to reason. 3. Models have not internalized mechanisms for human subjective heterogeneity. Humans react very differently to the same event—AI tends toward one standardized interpretation, as if everyone should feel the same way.

An overestimated capability

The paper's core conclusion: current emotion-prediction metrics likely overestimate LLMs' true emotion understanding. A 90% accuracy score may just mean the AI is good at matching labels, not understanding emotional processes. A label-matching chatbot told "I got promoted but I'm anxious" will spit out a congratulation template; an AI that understands appraisal might respond: "That's understandable—new challenges bring both excitement and worry. Try listing the three things about the new role that worry you most, then think through responses one by one." The former recites emotion; the latter understands its causes.

How far is true understanding?

The challenge isn't distinguishing happy from sad—low-level label matching is already solved. It's understanding why the same event produces different emotions in different people, and the cognitive appraisal beneath. Next time an AI seems empathetic, ask: does it really understand you, or is it matching a template it learned?

References

1. Sun, Z., Xu, H., Uusberg, A., Gross, J. J., Slovak, P., & He, Y. (2026). *CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning*. arXiv:2605.17176. 2. Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. In *Appraisal processes in emotion* (pp. 92-120). Oxford University Press. 3. Gross, J. J. (2015). *Emotion regulation: Current status and future prospects*. Psychological Inquiry, 26(1), 1-26. 4. Skowron, M., et al. (2018). Affect-based models for conversational AI. *NeurIPS Workshop on Affect Computing*. 5. Poria, S., et al. (2019). MNER: Multimodal emotion recognition. *Speech Communication*, 113, 63-75.