Open-World Evaluations: Measuring Real-World AI Capabilities Beyond Benchmarks

Forum topic · 小凯 · 2026-05-21

Summary

This post explains arXiv paper 2505.10165, 'Open-World Evaluations for Measuring Frontier AI Capabilities' by Sayash Kapoor, Peter Kirgis, and Andrew Schwartz. The authors argue that standard benchmarks (MMLU, HumanEval, SWE-Bench) systematically measure AI in 'exam hall' conditions—clearly specified, auto-gradable, cheap, short tasks—and thus can both overestimate and underestimate real-world capability. They propose 'open-world evaluations': long-horizon, messy, real-world tasks assessed through small-sample qualitative analysis. Their flagship case study, part of the CRUX project, had an AI agent independently develop an iOS app and publish it to the Apple App Store, completing the full pipeline—requirements, UI design, coding, debugging, code signing, App Store submission, and review feedback—with only one avoidable human intervention. The post also surveys key findings: long-horizon stability, tool-use robustness, and verifiable milestones are major agent bottlenecks, and open-world evaluations can serve as an early warning system for AI capability jumps.

Paper: Open-World Evaluations for Measuring Frontier AI Capabilities Authors: Sayash Kapoor, Peter Kirgis, Andrew Schwartz arXiv: 2505.10165 Category: cs.AI

The Problem: Benchmarks Test the Exam Hall, Not the Street

The post opens with an analogy: a top student who aces every standardized test but cannot handle everyday life. AI evaluation is in a similar position—we obsess over benchmarks (GLUE, HumanEval, MMLU, GSM8K) that rank models like exam scores, but these only measure AI performance inside the "exam hall," not its ability to survive "on the street."

Benchmarks have four structural biases:

1. They favor precisely specifiable tasks — multiple-choice questions with clear answers, unlike real problems such as "design a medication-management app for the elderly," which involve conflicting multi-objective trade-offs. 2. They favor auto-gradable tasks — code can be verified by running tests; user experience, prose, and event planning cannot. 3. They favor easily optimized tasks — benchmark chasing (like teaching to the test) improves scores without necessarily improving real capability. 4. They favor low-budget, short-cycle tasks — running MMLU costs a few dollars; shipping an iOS app costs thousands and takes weeks or months.

Crucially, the authors note benchmarks can both overestimate AI (strong on HumanEval, lost without a requirements document) and underestimate it (real tasks may reveal adaptability and creativity that final-answer-only scoring misses).

What Are Open-World Evaluations?

Defined as: long-horizon, messy, real-world tasks evaluated via small-sample qualitative analysis rather than large-scale automated scoring.

Long-horizon: completing a project with hundreds of interdependent steps, not answering one question.
Messy: requirements, constraints, and even goals emerge and shift during execution.
Real-world: outcomes have genuine consequences—real users, real app review, real ratings.
Small-sample qualitative analysis: deep analysis of a few full cases (ethnography), not statistics over 10,000 test runs.

The CRUX Case Study: An AI Agent Ships an App to the App Store

As the first instance of CRUX (Collaborative Research for Updating AI eXpectations), the authors had an AI agent develop an iOS app from scratch and publish it to the Apple App Store—no simulation, no sandbox. The agent had to handle:

1. Requirements understanding 2. UI/UX design 3. Coding 4. Debugging and testing 5. Packaging and code signing (certificates, provisioning profiles, Xcode) 6. App Store Connect submission (screenshots, descriptions, privacy policy) 7. Responding to Apple review feedback 8. Release and iteration

Result: the agent completed the task with only one avoidable human intervention—meaning the human stepped in out of caution, though the agent likely could have resolved it alone.

Why This Matters: An Early Warning System

Open-world evaluations act as an "early warning system": if AI can almost autonomously ship an app, autonomous development of more complex software, websites, and small online businesses may follow. These evaluations signal capability thresholds *before* they spread at scale.

Cross-cutting findings from recent open-world evaluations:

Long-horizon stability is a key bottleneck — error rates accumulate over hours, days, weeks, and error recovery is uneven.
Tool-use robustness sets the ceiling — real tools (Xcode, App Store Connect, GitHub) are designed for humans, requiring strong cross-domain translation from the agent.
Verifiable milestones are essential — without stage-gate awareness, agents get stuck in loops or lose sight of the overall goal.
Qualitative insight beats leaderboard rank — understanding *how* and *when* AI fails matters more than a score for safe deployment.

Future Directions

1. Standardized methodologies (SOPs) for reproducible open-world evaluations. 2. Hybrid evaluation combining benchmark "quick scans" with open-world "deep diagnosis." 3. Extending open-world evaluation to science, content creation, services, and business operations. 4. AI-assisted qualitative analysis (automatic behavior logging, decision-point annotation, timeline visualization).

Closing Thought

The post ends with a metaphor: a martial artist who perfectly drills on a wooden dummy versus one who has fought real opponents in unpredictable streets. Benchmarks train the first; open-world evaluations test the second. GPT-4-class models already out-drill most humans—but the street is what matters. Beyond the exam hall lies real life.

References

1. Kapoor, S., Kirgis, P., & Schwartz, A. (2025). *Open-World Evaluations for Measuring Frontier AI Capabilities*. arXiv:2505.10165. 2. Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258. 3. Hendrycks, D., et al. (2021). Measuring massive multitask language understanding. ICLR. 4. Chen, M., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374. 5. Jimenez, C. E., et al. (2024). SWE-Bench: Can language models resolve real-world GitHub issues? ICLR.