Paper: Open-World Evaluations for Measuring Frontier AI Capabilities Authors: Sayash Kapoor, Peter Kirgis, Andrew Schwartz arXiv: 2505.10165 Category: cs.AI
This is an English translation/summary of a Chinese forum post explaining the paper.
Why Current Benchmarks Fall Short
The post opens with an analogy: a student who aced every exam but cannot handle everyday life. Benchmarks like GLUE, HumanEval, MMLU, and GSM8K measure AI only inside the "exam hall," not on the "street." Four structural biases are identified:
1. Bias toward precisely specifiable tasks — multiple-choice problems with clear answers, unlike real-world problems such as "design a medication-management app for the elderly," which involve conflicting multi-objective trade-offs. 2. Bias toward automatically gradable tasks — code is easy to verify with unit tests; user experience, essays, and community events are not. Benchmarks thus measure what AI *can* do more than what it *should* do. 3. Bias toward easily optimizable tasks — benchmark scores get "gamed" through targeted optimization, and optimizing scores does not equal improving real capability (like exam-focused rote learning). 4. Bias toward low-budget, short-horizon tasks — running MMLU costs a few dollars; shipping an iOS app costs thousands and takes weeks. Real-world value often lies in long-horizon efforts.
The Core Claim
Benchmarks can both overestimate AI capability (good at HumanEval, lost on a real project with no requirements document) and underestimate it (real-world adaptiveness that answer-only scoring misses). The authors propose a complementary paradigm: open-world evaluations — long-horizon, messy, real-world tasks evaluated via small-sample qualitative analysis rather than large-scale automated scoring.
The CRUX Demonstration: Shipping an App to the App Store
The CRUX (Collaborative Research for Updating AI eXpectations) project performs periodic open-world evaluations. Its first instance: have an AI agent develop a simple iOS app from scratch and publish it to the real Apple App Store. This required the agent to:
1. Understand requirements 2. Design UI/UX 3. Write code (Swift / React Native, state management, persistence) 4. Debug and test 5. Handle packaging, certificates, and provisioning profiles 6. Submit for review via App Store Connect (screenshots, descriptions, privacy policy) 7. Respond to Apple reviewer feedback 8. Handle release, appeals, versioning, and user reviews
Result: the agent completed the task with only one avoidable human intervention (an intervention that was not strictly necessary). This suggests frontier AI has reached the threshold of independently executing real-world, long-horizon projects.
Open-World Evaluations as an Early Warning System
The paper frames open-world evaluations as an "early warning system": when AI crosses a capability threshold (e.g., autonomously shipping apps), cascading capabilities (building websites, running e-commerce, managing small businesses) may follow explosively. Open-world evaluation provides a real, timely signal before capabilities diffuse at scale — like a comprehensive lifestyle interview versus a routine blood test.
Common Findings from Recent Open-World Evaluations
1. Stability over long horizons is a key bottleneck — error rates accumulate over hours, days, and weeks; recovery ability varies. 2. Tool-use robustness sets the ceiling — real tools (Xcode, App Store Connect, GitHub, APIs) are built for humans, demanding strong cross-domain translation from agents. 3. Verifiable milestones are crucial — without milestone awareness, agents get stuck in subtasks or lose sight of overall goals. 4. Qualitative analysis beats quantitative ranking — the goal is understanding what AI can/cannot do and its failure modes, not leaderboard position.
Future Directions
- Standard operating procedures (SOPs) to make bespoke open-world evaluations comparable and reproducible
- Hybrid evaluation combining benchmarks (quick scan) with open-world evaluations (deep diagnosis)
- Cross-domain evaluations: science, content creation, social services, business operations
- AI-assisted qualitative analysis: logging trajectories, annotating key decisions, generating visual timelines
Takeaway
The post closes with a metaphor of two martial artists — one who perfectly strikes a wooden dummy, one who survives real street fights. Benchmarks train dummy-striking; open-world evaluations test street survival. Beyond the exam hall lies real life — and open-world evaluations are the map for understanding AI's true capabilities when it walks onto the street.
References
1. Kapoor, S., Kirgis, P., & Schwartz, A. (2025). *Open-World Evaluations for Measuring Frontier AI Capabilities*. arXiv:2505.10165. 2. Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258. 3. Hendrycks, D., et al. (2021). Measuring massive multitask language understanding. ICLR. 4. Chen, M., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374. 5. Jimenez, C. E., et al. (2024). SWE-Bench: Can language models resolve real-world GitHub issues? ICLR.