Agents' Last Exam: Benchmark Scores vs. Real-World Capability
TL;DR: Agents' Last Exam (ALE), developed by UC Berkeley RDI with 250+ industry experts, reveals that top coding AI—including Claude Fable 5, which scores 80% on SWE-Bench—achieves nearly 0% full pass rates on 1,490+ real professional workflows. High benchmark scores do not equal the ability to do real work.
Key Information
- Paper: Agents' Last Exam (ALE), arXiv:2606.05405
- Institution: UC Berkeley RDI × 250+ industry experts
- Scale: 55 subdomains, 13 industry clusters, 1,490+ real task instances
- Website: https://agents-last-exam.org/
- Leaderboard: https://agents-last-exam.org/leaderboard
- For AI developers: Stop fixating on SWE-Bench/OSWorld. Agent architecture (planning, memory, tool use, error recovery) matters more than raw model capability.
- For enterprise buyers: Benchmark scores are reference only—run POCs on real workflows. Prioritize long-horizon stability over single-point success rates; domain adaptation (fine-tuning + RAG + workflow orchestration) may matter more than switching models.
- For researchers: ALE is a living benchmark; 13/55 subdomains are completely uncovered by existing benchmarks (Table 1 in the paper). Legal, finance, and life sciences may be the next breakthrough areas.
- Sun, Yiyou et al. "Agents' Last Exam." arXiv:2606.05405 (2026). UC Berkeley RDI.
- SWE-Bench Pro: https://www.swe-bench.com/
- OSWorld: https://os-world.github.io/
- Terminal-Bench: https://terminal-bench.com/
- Berkeley RDI "How We Broke Top AI Agent Benchmarks" (April 2026)
- Anthropic Claude Fable 5 Launch (June 9, 2026)
- Scale AI SEAL Leaderboard: https://scale.com/leaderboard
A Counterintuitive Finding
Claude Fable 5's benchmark scores look impressive:
| Benchmark | Claude Fable 5 | Industry comparison | |-----------|---------------|---------------------| | SWE-Bench Pro | 80.3% | GPT-5.5: 58.6% | | Terminal-Bench 2.1 | 88.0% | Codex CLI: 83.4% | | OSWorld-Verified | 85.0% | GPT-5.5: 78.7% | | Humanity's Last Exam (with tools) | 64.5% | GPT-5.5: 52.2% |
But on ALE, its full pass rate is near 0%. This is not a Fable 5-specific failure: 24 of 27 top AI configurations scored 0% full pass rate across 1,490 real workflows, including Claude Opus 4.7 and GPT-5.5 combinations.
The paper states: "across mainstream harness and backbone configurations, the average full pass rate is below 1%" (later updated to 2.6%).
The strongest configuration (Codex + GPT-5.5) hits 82% on Terminal-Bench, but below 50% on ALE's easiest tier and below 10% on the hardest.
Why Current Benchmarks Fail to Reflect Real Ability
The paper identifies an unacceptable trade-off between realism, breadth, and verifiability in existing benchmarks:
| Dimension | Current benchmarks | ALE's solution | |-----------|-------------------|----------------| | Realism | Simplified environments or Q&A | Real VMs + real professional software + real data files | | Breadth | Single domains (code, web, desktop) | 55 subdomains, 13 industry clusters based on US SOC 2018 classification | | Verifiability | Human or subjective judgment | Structured deliverables + deterministic automatic scoring |
Case studies of benchmark inflation
1. SWE-Bench Pro scaffold bubble: Fable 5's 80.3% was achieved with Anthropic's own scaffold. On Scale AI's standardized SEAL leaderboard (https://scale.com/leaderboard), GPT-5.4 xHigh scored only 59.1% and Claude Opus 4.6 thinking only 51.9%. The same model can vary 20–30 points depending on the scaffold. 2. Berkeley broke its own benchmarks: In April 2026, Berkeley RDI showed that single-character modifications could achieve 100% scores on some top benchmarks without solving the tasks—890 tasks "passed" with one character. 3. OSWorld's lesson: OSWorld is considered hard to game (human baseline ~72–75%, Coasty agent 82%), but it only measures GUI operations. ALE = OSWorld + Terminal-Bench + cross-tool orchestration + end-to-end delivery + domain expertise—a superset.
What ALE Tests
Three hard admission criteria
1. Representativeness — must match real professional practice using the actual software experts use. Bad: "add a filter in DaVinci"; good: "composite a running cheetah into another racing video" (tracking, masking, compositing, color matching). 2. Complexity — end-to-end deliverable work that takes experts hours to weeks, not single operations. 3. Verifiability — deterministic checks. Bad: "design an RPG with monsters"; good: "recreate mota.exe in RPGMaker XP" (map geometry, character stats, and event states are automatically comparable).
Coverage: 55 subdomains, 13 industries
Domains are selected based on **US federal SOC 2018 + O*NET occupational classifications, covering all software-mediated professional workflows:
| Top-level domain | Example subdomains | |------------------|--------------------| | Architecture & Engineering | Civil, mechanical, electrical, architectural engineering | | Arts & Media | Graphic design, video editing, 3D modeling, music production | | Business & Finance | Accounting, financial analysis, market research, investment banking | | Computer & Mathematical | Software development, data analysis, cybersecurity, algorithms | | Education | Curriculum design, ed-tech, academic research | | Legal | Contract review, case study, compliance analysis | | Life Sciences | Bioinformatics, medical imaging, drug discovery | | Manufacturing | Supply chain, quality control, production planning |
Key numbers: 1,490+ implemented task instances; all 55 subdomains covered; 13 top-level industry clusters; 250+ industry experts involved.
Tasks are not crowdsourced—experts submit real completed projects, passing five gates: expert submission → initial screening → engineering implementation → engineer dry-run → expert committee final review.
Why AI Agents Fail Collectively
1. Long-horizon capability gap: SWE-Bench tasks modify ~4.1 files on average but remain single-point tasks; OSWorld tasks take minutes. ALE tasks require hours to weeks of work, multi-software collaboration (CLI + GUI + browser + professional tools), professional judgment, and complex deliverables (reports, designs, models, media files). 2. Cross-tool orchestration is the fatal bottleneck: The paper highlights the Generalist Computer-Use Agent (GCUA) requirement—visual perception, code execution, tool use, and long-range planning simultaneously. Current agents perform each step individually but collapse when chaining them. 3. Domain expertise gap: Real work requires domain knowledge (structural safety for civil engineers, tax rules for accountants, color matching for video editors) that generic coding cannot cover.
The Bitter Leaderboard Reality
| Difficulty tier | Average full pass rate | |----------------|------------------------| | Easiest | < 50% (best: Codex + GPT-5.5) | | Medium | ~10–20% | | Hardest | < 1% (updated to 2.6%) |
ALE's task surface covers GUI-only benchmarks (OSWorld) and CLI-only benchmarks (Terminal-Bench) while adding long-horizon, cross-tool, end-to-end delivery requirements.
Core Insight: Benchmark Success ≠ GDP Impact
The paper's diagnosis—the "Utility Problem":
> "AI systems have cleared one celebrated benchmark after another... Yet by the metric that ultimately matters, economic output, the broader impact has remained surprisingly muted."
> "ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP-relevant impact."
Implications
Limitations and Future Directions
The authors honestly list limitations:
1. No physical world: ALE covers only software-mediated (non-physical) workflows, not robotics or physical manufacturing. 2. Verification difficulty: Objective scoring remains hard for creative writing and design aesthetics. 3. Static sandbox: Real work environments change (software updates, rule changes); ALE's sandbox is static. 4. No safety evaluation: ALE does not test agent safety boundaries in real environments.
Future work includes onboarding new industries and workflows, integrating safety/privacy/ethics evaluation, and exploring human-in-the-loop assistive modes.
Conclusion
Claude Fable 5's 80% on SWE-Bench vs. ~0% on ALE is not because the model got worse—it is because ALE tests "completing whole work," not "fixing a single bug." The real bottlenecks are long-horizon planning and execution, cross-tool orchestration, domain expertise integration, and error recovery.
> AI's next milestone is not another 5-point benchmark bump, but turning full pass rates from 0% into a meaningful number.**