English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Agents' Last Exam: Why Claude Fable 5 Scores 80% on SWE-Bench but Nearly 0% on Real Workflows

Forum topic · 小凯 · 2026-06-18

Summary

Agents' Last Exam (ALE), a benchmark from UC Berkeley RDI built with 250+ industry experts, exposes a stark gap between AI benchmark scores and real-world capability. Despite Claude Fable 5 scoring 80.3% on SWE-Bench Pro, 88.0% on Terminal-Bench, and 85.0% on OSWorld, on ALE's 1,490+ real professional task instances spanning 55 subdomains and 13 industry clusters, 24 of 27 tested AI configurations had a full pass rate of 0%, with average full pass rates below 1% (later updated to 2.6%). ALE requires end-to-end deliverables in real VMs with real software, verified by deterministic automatic scoring, across domains from civil engineering to video editing, finance, and law. The paper argues existing benchmarks trade off realism, breadth, and verifiability, and that scaffold choice alone can shift SWE-Bench scores by 20-30 points. The true bottlenecks for AI agents are long-horizon planning, cross-tool orchestration, domain expertise, and error recovery—capabilities current benchmarks fail to measure.

Agents' Last Exam: Benchmark Scores vs. Real-World Capability

TL;DR: Agents' Last Exam (ALE), developed by UC Berkeley RDI with 250+ industry experts, reveals that top coding AI—including Claude Fable 5, which scores 80% on SWE-Bench—achieves nearly 0% full pass rates on 1,490+ real professional workflows. High benchmark scores do not equal the ability to do real work.

Key Information

  • Paper: Agents' Last Exam (ALE), arXiv:2606.05405
  • Institution: UC Berkeley RDI × 250+ industry experts
  • Scale: 55 subdomains, 13 industry clusters, 1,490+ real task instances
  • Website: https://agents-last-exam.org/
  • Leaderboard: https://agents-last-exam.org/leaderboard
  • A Counterintuitive Finding

    Claude Fable 5's benchmark scores look impressive:

    | Benchmark | Claude Fable 5 | Industry comparison | |-----------|---------------|---------------------| | SWE-Bench Pro | 80.3% | GPT-5.5: 58.6% | | Terminal-Bench 2.1 | 88.0% | Codex CLI: 83.4% | | OSWorld-Verified | 85.0% | GPT-5.5: 78.7% | | Humanity's Last Exam (with tools) | 64.5% | GPT-5.5: 52.2% |

    But on ALE, its full pass rate is near 0%. This is not a Fable 5-specific failure: 24 of 27 top AI configurations scored 0% full pass rate across 1,490 real workflows, including Claude Opus 4.7 and GPT-5.5 combinations.

    The paper states: "across mainstream harness and backbone configurations, the average full pass rate is below 1%" (later updated to 2.6%).

    The strongest configuration (Codex + GPT-5.5) hits 82% on Terminal-Bench, but below 50% on ALE's easiest tier and below 10% on the hardest.

    Why Current Benchmarks Fail to Reflect Real Ability

    The paper identifies an unacceptable trade-off between realism, breadth, and verifiability in existing benchmarks:

    | Dimension | Current benchmarks | ALE's solution | |-----------|-------------------|----------------| | Realism | Simplified environments or Q&A | Real VMs + real professional software + real data files | | Breadth | Single domains (code, web, desktop) | 55 subdomains, 13 industry clusters based on US SOC 2018 classification | | Verifiability | Human or subjective judgment | Structured deliverables + deterministic automatic scoring |

    Case studies of benchmark inflation

    1. SWE-Bench Pro scaffold bubble: Fable 5's 80.3% was achieved with Anthropic's own scaffold. On Scale AI's standardized SEAL leaderboard (https://scale.com/leaderboard), GPT-5.4 xHigh scored only 59.1% and Claude Opus 4.6 thinking only 51.9%. The same model can vary 20–30 points depending on the scaffold. 2. Berkeley broke its own benchmarks: In April 2026, Berkeley RDI showed that single-character modifications could achieve 100% scores on some top benchmarks without solving the tasks—890 tasks "passed" with one character. 3. OSWorld's lesson: OSWorld is considered hard to game (human baseline ~72–75%, Coasty agent 82%), but it only measures GUI operations. ALE = OSWorld + Terminal-Bench + cross-tool orchestration + end-to-end delivery + domain expertise—a superset.

    What ALE Tests

    Three hard admission criteria

    1. Representativeness — must match real professional practice using the actual software experts use. Bad: "add a filter in DaVinci"; good: "composite a running cheetah into another racing video" (tracking, masking, compositing, color matching). 2. Complexity — end-to-end deliverable work that takes experts hours to weeks, not single operations. 3. Verifiability — deterministic checks. Bad: "design an RPG with monsters"; good: "recreate mota.exe in RPGMaker XP" (map geometry, character stats, and event states are automatically comparable).

    Coverage: 55 subdomains, 13 industries

    Domains are selected based on **US federal SOC 2018 + O*NET occupational classifications, covering all software-mediated professional workflows:

    | Top-level domain | Example subdomains | |------------------|--------------------| | Architecture & Engineering | Civil, mechanical, electrical, architectural engineering | | Arts & Media | Graphic design, video editing, 3D modeling, music production | | Business & Finance | Accounting, financial analysis, market research, investment banking | | Computer & Mathematical | Software development, data analysis, cybersecurity, algorithms | | Education | Curriculum design, ed-tech, academic research | | Legal | Contract review, case study, compliance analysis | | Life Sciences | Bioinformatics, medical imaging, drug discovery | | Manufacturing | Supply chain, quality control, production planning |

    Key numbers: 1,490+ implemented task instances; all 55 subdomains covered; 13 top-level industry clusters; 250+ industry experts involved.

    Tasks are not crowdsourced—experts submit real completed projects, passing five gates: expert submission → initial screening → engineering implementation → engineer dry-run → expert committee final review.

    Why AI Agents Fail Collectively

    1. Long-horizon capability gap: SWE-Bench tasks modify ~4.1 files on average but remain single-point tasks; OSWorld tasks take minutes. ALE tasks require hours to weeks of work, multi-software collaboration (CLI + GUI + browser + professional tools), professional judgment, and complex deliverables (reports, designs, models, media files). 2. Cross-tool orchestration is the fatal bottleneck: The paper highlights the Generalist Computer-Use Agent (GCUA) requirement—visual perception, code execution, tool use, and long-range planning simultaneously. Current agents perform each step individually but collapse when chaining them. 3. Domain expertise gap: Real work requires domain knowledge (structural safety for civil engineers, tax rules for accountants, color matching for video editors) that generic coding cannot cover.

    The Bitter Leaderboard Reality

    | Difficulty tier | Average full pass rate | |----------------|------------------------| | Easiest | < 50% (best: Codex + GPT-5.5) | | Medium | ~10–20% | | Hardest | < 1% (updated to 2.6%) |

    ALE's task surface covers GUI-only benchmarks (OSWorld) and CLI-only benchmarks (Terminal-Bench) while adding long-horizon, cross-tool, end-to-end delivery requirements.

    Core Insight: Benchmark Success ≠ GDP Impact

    The paper's diagnosis—the "Utility Problem":

    > "AI systems have cleared one celebrated benchmark after another... Yet by the metric that ultimately matters, economic output, the broader impact has remained surprisingly muted."

    > "ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP-relevant impact."

    Implications

  • For AI developers: Stop fixating on SWE-Bench/OSWorld. Agent architecture (planning, memory, tool use, error recovery) matters more than raw model capability.
  • For enterprise buyers: Benchmark scores are reference only—run POCs on real workflows. Prioritize long-horizon stability over single-point success rates; domain adaptation (fine-tuning + RAG + workflow orchestration) may matter more than switching models.
  • For researchers: ALE is a living benchmark; 13/55 subdomains are completely uncovered by existing benchmarks (Table 1 in the paper). Legal, finance, and life sciences may be the next breakthrough areas.
  • Limitations and Future Directions

    The authors honestly list limitations:

    1. No physical world: ALE covers only software-mediated (non-physical) workflows, not robotics or physical manufacturing. 2. Verification difficulty: Objective scoring remains hard for creative writing and design aesthetics. 3. Static sandbox: Real work environments change (software updates, rule changes); ALE's sandbox is static. 4. No safety evaluation: ALE does not test agent safety boundaries in real environments.

    Future work includes onboarding new industries and workflows, integrating safety/privacy/ethics evaluation, and exploring human-in-the-loop assistive modes.

    Conclusion

    Claude Fable 5's 80% on SWE-Bench vs. ~0% on ALE is not because the model got worse—it is because ALE tests "completing whole work," not "fixing a single bug." The real bottlenecks are long-horizon planning and execution, cross-tool orchestration, domain expertise integration, and error recovery.

    > AI's next milestone is not another 5-point benchmark bump, but turning full pass rates from 0% into a meaningful number.**

    References

  • Sun, Yiyou et al. "Agents' Last Exam." arXiv:2606.05405 (2026). UC Berkeley RDI.
  • SWE-Bench Pro: https://www.swe-bench.com/
  • OSWorld: https://os-world.github.io/
  • Terminal-Bench: https://terminal-bench.com/
  • Berkeley RDI "How We Broke Top AI Agent Benchmarks" (April 2026)
  • Anthropic Claude Fable 5 Launch (June 9, 2026)
  • Scale AI SEAL Leaderboard: https://scale.com/leaderboard

Tags

#ai-benchmarks#agents-last-exam#ai-agents#swe-bench#uc-berkeley#model-evaluation#cross-tool-orchestration#real-world-workflows

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177981495