CL-bench Life: Frontier LLMs Score Only 13.8% on Real-Life Context Tasks

Forum topic · 小凯 · 2026-06-08

Summary

CL-bench Life, a benchmark from the Tencent Hunyuan team and Fudan University, evaluates whether large language models can learn from real-life contexts such as group chats, shopping lists, fitness logs, and landlord-tenant conversations. Across 405 fully human-annotated tasks with 5,348 binary verification criteria, spanning three categories (communication and social interaction, fragmented information and revisions, and behavioral records and activity traces), the average task-solving rate for ten frontier models is only 13.8%, with the best model, GPT-5.4, reaching just 19.3%. The hardest subcategory is self-tracking trajectories (fitness and health logs), where the best score is 10.4%. Key findings: performance is not strongly correlated with context length; reasoning helps but with diminishing returns; the dominant failure mode is Context-Misused (models read but misunderstand context) rather than Context-Ignored; role confusion and speaker attribution errors dominate in group chats; and removing context crashes GPT-5.4 from 19.3% to 1.7%, confirming tasks require genuine in-context learning. The benchmark highlights fundamental gaps in social role understanding, temporal reasoning, sparse pattern aggregation, and precise constraint following.

CL-bench Life Deep Dive: Why AI Collectively Fails at Real-Life Contexts

> Paper: CL-bench Life: Can Language Models Learn from Real-Life Context? > Authors: Tencent Hunyuan team × Fudan University > arXiv: 2604.27043 | April 2026 > Project: https://github.com/Tencent-Hunyuan/CL-bench

The Core Question

Existing benchmarks test professional domains—code, finance, scientific papers. But what about real-life context: group chats, shopping lists, fitness logs, three years of landlord-tenant disputes? This is the real battlefield for AI assistants.

CL-bench Life's answer: the average task-solving rate is 13.8%. The strongest model, GPT-5.4, reaches only 19.3%. On a 100-point scale, no model comes close to passing.

Dataset Construction

405 tasks, 5,348 binary verification criteria, 3 categories, 9 subcategories

| Category | Subcategories | Count | Core challenge | |---|---|---|---| | Communication & social interaction | Private conversations; group chats & meeting logs; community interactions | 45 each | Topic jumps, implied emotion, multi-party references, role confusion, nested replies, alias coreference | | Fragmented information & revisions | Personal info fragments; public info fragments; creation & revision history | 45 each | Loosely organized notes/lists, RSS/news aggregation, version evolution, conflict resolution | | Behavioral records & activity traces | Game logs; digital footprints & daily records; self-tracking trajectories | 45 each | Strategy analysis, browsing/transaction/location patterns, long-term fitness/health trend inference |

Design principles

Fully human-annotated: each task takes ~13 hours of expert effort
Self-contained context: no external retrieval needed
Multi-turn: 59.8% of instances include multi-turn dialogue history
Strict evaluation: all rubric items must be satisfied; GPT-5.1 as judge with >90% human-verified accuracy

Results: Ten Models Collectively Fail

| Rank | Model | Overall | Communication | Fragmented | Behavioral | |---|---|---|---|---|---| | 1 | GPT-5.4 | 19.3% | 22.2% | 15.8% | 20.0% | | 2 | Claude Opus 4.6 | ~17% | ~19% | ~16% | ~17% | | 3 | Gemini 3.1 Pro | ~16% | ~17% | ~15% | ~18% | | 4 | Hy3 preview | ~15% | ~16% | ~14% | ~15% | | 5 | Seed 2.0 Pro | ~14% | ~15% | ~13% | ~14% | | 6 | Kimi K2.5 | ~13% | ~14% | ~12% | ~13% | | 7 | Qwen 3.5 Plus | ~13% | ~13% | ~12% | ~14% | | 8 | Grok 4.20 | ~12% | ~13% | ~11% | ~12% | | 9 | DeepSeek V3.2 | ~10% | ~11% | ~9% | ~10% | | 10 | MiniMax M2.5 | ~8% | ~9% | ~7% | ~8% |

Hardest subcategory: self-tracking trajectories (fitness/health logs)—best score 10.4% (GPT-5.4), most models below 6%. The data is sparse, weakly structured, and requires aggregating months of tiny events to infer implicit patterns.

Key Findings

1. Context length is not strongly correlated with performance

Without reasoning, some models degrade with length—but with reasoning, GPT-5.4 achieves its highest score (23.1%) on 32K+ long contexts, and Claude Opus 4.6 recovers on the longest texts. The problem is not "can't finish reading" but "doesn't understand."

2. Reasoning helps, with diminishing returns

Gains are largest for behavioral records (scattered evidence needs restructuring); limited for communication tasks.
GPT-5.4: ~15% (low reasoning) → ~22% (ultra-high reasoning), but at multiplied token cost.
More reasoning tokens ≠ better answers. The value of reasoning lies in organizing context more effectively, producing leaner, more focused answers.

3. Main cause of death: Context-Misused, not Context-Ignored

Context-Misused (highest): model reads the context but misunderstands it or fails to integrate information across passages.
Context-Ignored (second): key information never used.
Format-Error (lower); Refusal (lowest).

The model "looked but didn't understand"—good at surface cues, poor at underlying relational reasoning.

4. Group chats are a disaster zone

Top error: Role Confusion—treating the channel creator as the leader and the actual decision-maker as a subordinate. Also frequent: speaker attribution errors, alias coreference failures, stale information. Example: in a Slack channel where Alice creates rules but Clara makes final decisions, Gemini-3.1-Pro inverted the entire hierarchy.

5. No-context ablation: tasks genuinely require context

With context, GPT-5.4: 19.3%; without: 1.7% (a -17.6pp drop). Tasks cannot be solved from parametric memory.

6. Test-time scaling is limited

Pass@k improves with k but saturates around k=8; Pass^k degrades sharply, with GPT-5.4 already dropping heavily at k=4. The bottleneck is the fundamental difficulty of real-life in-context learning, not insufficient exploration.

Methodology

Judge: GPT-5.1 (high reasoning), decoupled design—the judge sees only the answer and rubrics, not the context or task, avoiding contamination.
Strictness: all rubric items (~13.2 per task) must pass.
Cross-judge agreement: GPT-5.1 vs Claude-Opus-4.6: 93.2%, Cohen's κ = 0.724; three-judge panels: 93.0–94.5%, κ = 0.710–0.773.
Failure taxonomy: Context-Ignored, Context-Misused, Format-Error, Refusal.

Case Study: Finding the Biggest Financial Leak

Task: given four months of bank transactions, identify the recurring spending pattern most blocking savings growth, cite 3–5 specific transactions, compute monthly and four-month totals, and compare with a March savings goal.

GPT-5.4 (high reasoning):

✅ Correctly identified convenience-store spending as the main leak
✅ Located the March SoFi savings goal
✅ Avoided distractors (gambling, subscriptions)
❌ But all monthly totals were computed wrong
❌ Misclassified convenience-store ATM withdrawals as spending
❌ Introduced unsupported speculation despite explicit instructions not to

Implication: models recognize patterns but struggle with precise computation and strict constraint-following—be cautious trusting AI with exact numbers in financial analysis.

Why This Matters

1. Fills a gap: prior benchmarks cover professional domains; daily assistant scenarios (organizing chats, analyzing habits, tracking health) were never systematically evaluated. 2. Quantifies the difficulty of "life sense": not a length or knowledge problem—the messy, fragmented, socially grounded nature of real-life context is a fundamental challenge. 3. Points to improvements: role understanding, temporal reasoning, sparse pattern recognition, precise constraint following. 4. Explains user experience: why AI "seems to understand but doesn't quite" in everyday scenarios.

The paper's metaphor: existing benchmarks test advanced math—where AI scores 90; CL-bench Life tests life common sense—where AI scores 13.

Summary Table

| Dimension | Data / conclusion | |---|---| | Dataset | 405 tasks, 5,348 rubrics, 3 categories, 9 subcategories | | Best model | GPT-5.4, 19.3% | | Average | 13.8% | | Hardest subcategory | Self-tracking trajectories, best 10.4% | | Main failure mode | Context-Misused | | Group chat errors | Role confusion (highest), speaker attribution | | Length effect | No strong correlation; GPT-5.4 peaks at 23.1% on 32K+ | | Reasoning | Helps but diminishing returns | | No-context ablation | 19.3% → 1.7% | | Test-time scaling | Pass@k saturates at k=8 |

One-sentence takeaway: frontier AI cannot pass even a basic bar in real-life scenarios—the problem is not reading too little, but "reading without understanding."

References:

Paper: arXiv:2604.27043 [cs.CL]
Project: https://github.com/Tencent-Hunyuan/CL-bench
Authors: Tencent Hunyuan team × Fudan University