HEART-Bench: Giving AI a "Psychological Exam" — 11 Virtual Personas, 1000 Memories, 673 Questions
> Source: HEART-Bench: Do LLM Agents Exhibit Human-like Psychology?, Weihan Peng, Chenxu Zhang, Qianao Wang, Yuling Shi, Heng Lian et al., https://arxiv.org/abs/2605.30058
---
Have you ever wondered: if you ask an LLM to play a "high-neuroticism, low-extraversion" person, can it stay consistent across all scenarios?
Not the one-dimensional acting where you say "play an anxious person" and it adds exclamation marks to every sentence. Instead — give it a complete life history, then see whether its behavioral decisions across workplace, romance, social, and crisis scenarios actually follow that person's personality logic.
That is exactly what HEART-Bench tests.
Not Just "Acting" — But "Becoming"
Existing LLM personality tests are mostly simple: describe a personality in a prompt, ask a few questions, check whether answers fit. The problem: such tests only measure "acting ability," not "personality consistency" — a model might act introverted in question one and forget who it is by question five.
HEART-Bench takes a completely different approach. It builds 11 complete human characters, each with:
1. Orthogonal Big Five traits: openness, conscientiousness, extraversion, agreeableness, neuroticism — five dimensions varied independently to ensure essential differences between characters. 2. 1000 autobiographical episodic memories: not dry personality descriptions, but concrete, emotionally warm memories like "the first time I spoke in front of the whole school at age seven, my palms were drenched with sweat." These memories are distributed across life stages — childhood, adolescence, early adulthood, middle age — following developmental psychology theory. 3. Human validation: all character settings and questions were manually curated to ensure psychological plausibility.
The DIAMONDS Framework: Scanning Personality Across 8 Dimensions
Characters alone are not enough; you need test scenarios. HEART-Bench uses the psychological DIAMONDS taxonomy, classifying situations into 8 dimensions:
| Dimension | Meaning | Example scenario | |------|------|------| | Duty | Obligations | Completing tasks on time at work | | Intellect | Intellectual challenge | Deep thinking on complex problems | | Adversity | Adversity and stress | How one responds to failure | | Mating | Romance and intimacy | Reacting to a confession of love | | pOsitivity | Positive situations | Responding to an unexpected gift | | Negativity | Negative situations | Handling being misunderstood | | Deception | Deception and trust | Discovering a friend lied | | Sociality | Social interaction | How one behaves at a party |
Every character must make behavioral decisions in all 8 scenario categories, forming 673 multiple-choice questions, each human-verified.
What Is Measured? Not Knowledge — "Personality Consistency"
The core metric is not "how many correct answers" but: does a high-neuroticism person lean toward anxious responses under adversity more than a low-neuroticism one? Does a high-extraversion person act more proactively in social settings?
This resembles "construct validity" in psychology experiments — not surface scores, but whether behavioral patterns match theoretical expectations.
Why Does This Matter?
Current AI agent research focuses almost entirely on "task capability" — reasoning, planning, tool use. But a genuinely useful AI assistant needs not only task completion but also stable personality traits. You don't want an assistant that is warm today and cold tomorrow.
The deeper implication: if an LLM cannot maintain a consistent persona, it is unreliable in any scenario requiring long-term trust — psychological counseling, educational tutoring, customer service. The core question in these domains is not "can it do it" but "can it be trusted."
HEART-Bench sends a clear signal: we need not just smarter AI, but psychologically more "complete" AI. Measuring that completeness demands experiment design as rigorous as psychology — building full characters, designing systematic scenarios, and validating behavioral consistency.
673 questions is not many, but each is backed by personality theory, developmental psychology, and human validation. It brings psychological experimental methodology into AI evaluation — not making AI take a test, but giving AI a psychological exam.
---
Paper link: https://arxiv.org/abs/2605.30058