CL-bench Life: Why Frontier AI Models Fail at Real-Life Contexts

Forum topic · 小凯 · 2026-06-08

Summary

CL-bench Life, a benchmark from Tencent Hunyuan and Fudan University, tests whether large language models can learn from real-life contexts such as group chats, shopping lists, fitness logs, and financial records. Spanning 405 tasks across 3 categories and 9 subcategories, with 5,348 binary verification rubrics, the benchmark reveals a stark gap: frontier models average only 13.8% task resolution, with the best model (GPT-5.4) reaching just 19.3%. The hardest subcategory, self-tracking trajectories, tops out at 10.4%. Error analysis shows the dominant failure mode is Context-Misused—models read context but misunderstand it—rather than ignoring it. Performance is not strongly correlated with context length, and reasoning helps but with diminishing returns. Removing context drops GPT-5.4's score from 19.3% to 1.7%, confirming tasks genuinely require in-context learning. The paper identifies key weaknesses in role understanding, temporal reasoning, sparse pattern aggregation, and precise constraint following in everyday scenarios.

CL-bench Life: Can Language Models Learn from Real-Life Context?

> Paper: CL-bench Life: Can Language Models Learn from Real-Life Context? > Authors: Tencent Hunyuan team × Fudan University > arXiv: 2604.27043 | April 2026

1. The Core Question

Existing benchmarks test professional domains—code, finance, scientific papers. But what about real-life contexts: group chats, shopping lists, fitness logs, three-year landlord-tenant disputes? These are the actual battlegrounds for AI assistants.

CL-bench Life's answer: frontier LLMs average 13.8% task resolution in real-life scenarios. The strongest, GPT-5.4, reaches only 19.3%. None pass a 20% mark.

2. Dataset Construction: 405 Tasks, 5,348 Binary Rubrics

2.1 Three Categories, Nine Subcategories

| Category | Subcategory | Tasks | Core Challenge | |---|---|---|---| | Communication & Social Interaction | Private conversations | 45 | Topic jumps, implied emotion, personal expression | | | Group chats & meeting logs | 45 | Multi-party references, role confusion, scattered info | | | Community interactions | 45 | Nested replies, alias coreference, topic drift | | Fragmented Info & Revisions | Personal info fragments | 45 | Loosely organized notes, lists, bookmarks | | | Public info fragments | 45 | Aggregated RSS, news snippets, search results | | | Creation & revision history | 45 | Version evolution, intent inference, conflict resolution | | Behavioral Records & Activity Traces | Game logs | 45 | Strategy analysis, event sequences, rule inference | | | Digital footprints & daily records | 45 | Pattern recognition over browsing, transactions, locations | | | Self-tracking trajectories | 45 | Trend inference from long-term fitness/health data |

2.2 Key Design Principles

Fully human-annotated: average 13 hours of expert effort per task
Self-contained context: no external retrieval required
Multi-turn interaction: 59.8% of instances include multi-turn dialogue history
Strict evaluation: all rubrics must be satisfied; GPT-5.1 as judge with >90% human-verified accuracy

3. Results: Top Ten Models Collectively Fail

3.1 Overall Ranking (Reasoning, High setting)

| Rank | Model | Overall | Communication | Fragmented | Behavioral | |---|---|---|---|---|---| | 1 | GPT-5.4 | 19.3% | 22.2% | 15.8% | 20.0% | | 2 | Claude Opus 4.6 | ~17% | ~19% | ~16% | ~17% | | 3 | Gemini 3.1 Pro | ~16% | ~17% | ~15% | ~18% | | 4 | Hy3 preview | ~15% | ~16% | ~14% | ~15% | | 5 | Seed 2.0 Pro | ~14% | ~15% | ~13% | ~14% | | 6 | Kimi K2.5 | ~13% | ~14% | ~12% | ~13% | | 7 | Qwen 3.5 Plus | ~13% | ~13% | ~12% | ~14% | | 8 | Grok 4.20 | ~12% | ~13% | ~11% | ~12% | | 9 | DeepSeek V3.2 | ~10% | ~11% | ~9% | ~10% | | 10 | MiniMax M2.5 | ~8% | ~9% | ~7% | ~8% |

Average: 13.8%. No model exceeds 20%.

3.2 Hardest Subcategories

| Subcategory | Best Model | Best Score | |---|---|---| | Group chats & meeting logs | GPT-5.4 | 30.4% | | Game logs | GPT-5.4 | 30.4% | | Community interactions | Seed 2.0 Pro | 23.0% | | Private conversations | Claude Opus 4.6 | 20.7% | | Public info fragments | Claude Opus 4.6 | 20.0% | | Personal info fragments | Gemini 3.1 Pro | 20.0% | | Creation & revision history | Gemini 3.1 Pro | 18.5% | | Digital footprints | GPT-5.4 | 19.3% | | Self-tracking trajectories | GPT-5.4 | 10.4% |

Self-tracking trajectories (fitness logs, health records, learning progress) is the hardest: sparse, weakly structured data requiring aggregation of months of small events and inference of implicit patterns.

4. Key Findings

Finding 1: Length Is Not the Problem

In non-reasoning mode, some models degrade with length
But in reasoning mode, GPT-5.4 achieves its highest score (23.1%) on 32K+ contexts

Conclusion: the problem is not "can't read it all" but "can't understand it."

Finding 2: Reasoning Helps, With Diminishing Returns

| Dimension | Result | |---|---| | Behavioral records | Largest gains (requires reconstructing scattered evidence) | | Fragmented info | Uneven gains | | Communication | Limited gains | | GPT-5.4 | ~15% (low reasoning) → ~22% (ultra-high), at multiples of token cost | | Efficiency | More reasoning tokens ≠ better answers |

Finding 3: Main Failure Mode Is Context-Misused, Not Context-Ignored

| Failure mode | Prevalence | |---|---| | Context-Misused | Highest — read but misunderstood or incorrectly integrated | | Context-Ignored | Second | | Format-Error | Lower | | Refusal | Lowest |

Models grab surface cues but fail at underlying relational reasoning.

Finding 4: Group Chats Are a Disaster Zone

| Error type | Frequency | |---|---| | Role Confusion | Highest | | Speaker Attribution errors | High | | Alias Coreference failures | Medium | | Stale Information | Low |

Typical case: In a Slack channel, Alice creates the channel and sets rules while Clara makes the actual decisions. Gemini-3.1-Pro treated Alice as senior and Clara as subordinate—an inverted authority structure corrupting all downstream reporting-chain reasoning.

Finding 5: No-Context Ablation Confirms Genuine Context Dependence

| Setting | GPT-5.4 Overall | |---|---| | With context | 19.3% | | Without context | 1.7% |

Tasks cannot be solved via parametric memory.

Finding 6: Limited Inference-Time Scaling

Pass@k saturates at k=8; Pass^k (all k correct) drops sharply, with GPT-5.4 declining heavily by k=4. The bottleneck is the fundamental difficulty of real-life in-context learning, not insufficient exploration.

5. Technical Pipeline

1. Define scenarios of everyday context types 2. Build contexts from private, public, or newly created sources, scrubbed of sensitive data 3. Design tasks requiring reasoning, not simple needle-in-haystack retrieval 4. Write rubrics: binary Yes/No criteria, averaging 13.2 per task 5. Quality checks: iterative sampling and feedback

Evaluation: GPT-5.1 judge (high reasoning) in a decoupled design—the judge sees only the answer and rubrics, not the context or task. Cross-judge agreement: GPT-5.1 vs Claude-Opus-4.6 at 93.2% (Cohen's κ = 0.724); three-judge consistency 93.0–94.5% (κ = 0.710–0.773).

6. Case Study: Finding the Biggest Financial Leak

Task: identify the largest recurring spending pattern over four months of bank statements, cite 3–5 transactions, compute monthly and total amounts, and compare against a March SoFi savings goal.

GPT-5.4 (high reasoning):

✅ Correctly identified convenience-store spending as the main leak
✅ Located the March SoFi goal for comparison
✅ Avoided distractors (gambling, subscriptions)
❌ All monthly totals computed incorrectly
❌ Misclassified convenience-store ATM withdrawals as store purchases
❌ Introduced speculative claims explicitly forbidden by the prompt

Insight: models recognize patterns but struggle with precise calculation and strict constraint adherence—AI can be directionally right yet numerically wrong.

7. Why This Matters

Fills a gap: first comprehensive benchmark of everyday real-life contexts
Quantifies the difficulty: messy, fragmented, socially grounded context is a fundamental challenge—not a length or knowledge problem
Points to improvements needed: role understanding, temporal reasoning, sparse pattern aggregation, precise constraint following
For AI assistant developers: explains why users often feel AI "seems to get it, but not really"

The paper's metaphor: if existing benchmarks are the calculus exam, CL-bench Life is the life-skills exam—and AI scores 90 on calculus, 13 on life.

8. Summary Table

| Dimension | Result | |---|---| | Dataset | 405 tasks, 5,348 rubrics, 3 categories / 9 subcategories | | Best model | GPT-5.4, 19.3% | | Average | 13.8% | | Hardest subcategory | Self-tracking trajectories (best: 10.4%) | | Main failure mode | Context-Misused | | Group chat errors | Role confusion (top), speaker attribution | | Length impact | Weak correlation; GPT-5.4 peaks at 23.1% on 32K+ | | Reasoning | Helps but diminishing returns | | No-context ablation | 19.3% → 1.7% | | Inference-time scaling | Pass@k saturates at k=8 |

One-line takeaway: today's most advanced AI cannot pass even the lowest bar in real-life scenarios—not because context is too long, but because models read without understanding.

References:

Paper: arXiv:2604.27043 [cs.CL]
Project page: https://github.com/Tencent-Hunyuan/CL-bench
Authors: Tencent Hunyuan team × Fudan University