When every benchmark is approaching a perfect score, how do you measure what an AI can actually do? Eighteen researchers from Princeton and Stanford offer an answer: stop testing—give it a real task to do.
Paper Information
| Item | Detail | |------|--------| | Title | Open-World Evaluations for Measuring Frontier AI Capabilities | | Authors | Sayash Kapoor, Arvind Narayanan, et al. (18 researchers) | | Institutions | Princeton, Stanford, Johns Hopkins, Oxford, UW Madison, Microsoft Research, UK AISI, and others | | arXiv | 2605.20520 | | Submitted | May 19, 2026 | | Core claim | Benchmarks have systematically failed at measuring frontier AI capabilities—they can overestimate (via benchmark gaming) and underestimate (when models get stuck on CAPTCHAs rather than real capability gaps). The alternative is open-world evaluation: have AI attempt long, messy, unpredictable real-world tasks, then have a small group of experts carefully read its logs to see where it was smart and where it fell short. |
The story starts with an awkward fact: between 2025 and 2026, nearly every well-known AI benchmark—SWE-Bench, ARC-AGI, τ-bench, Terminal Bench—was pushed toward ceiling scores. Each time a benchmark was saturated, the community rushed out a harder successor, which was then quickly saturated too. An endless arms race in which increasingly capable AIs trampled every new wall flat.
The problem isn't that the walls are too low. The problem is that we keep using one ruler—"how many questions can you answer correctly in a clean, predictable environment"—to measure abilities that need to operate in the real world.
That's why researchers from Princeton, Stanford, Johns Hopkins, Microsoft Research, and the UK AI Safety Institute did something that looks a bit crazy: they had an AI develop an iOS app and publish it on the Apple App Store.
Not a simulation. Not a sandbox. Not a benchmark question. Real Apple. Real reviewers. Really live.
The Hidden Crack: When Every Exam Becomes a Grind
The paper spends considerable space on a fundamental methodological dilemma. Benchmarks require every task to be: precisely specifiable, automatically gradable, easily optimizable against, and cheap and quick. Together these conditions produce a compressed sketch. Real capability looks like an oil painting.
When SWE-Bench reports 57.7%, how much is genuine software engineering mastery, and how much is learning task-specific solutions? Conversely, when an AI fails a whole test because it gets stuck on a CAPTCHA, the score records a 0—even if it could have written the code perfectly.
In 2010, mathematician David Donoho called benchmarks machine learning's "secret sauce" of the past 50 years. But the paper's argument is surgical: when the target becomes too clear, a crack opens between "aiming at the target" and "learning the skill." Open-world evaluation is an attempt to work inside that crack.
CRUX #1: Letting AI Face Apple Review
CRUX stands for Collaborative Research for Updating AI eXpectations—a platform letting independent researchers regularly and systematically run real-world exams. The first experiment's task:
"Develop and release a simple iOS application."
The AI's tools: a macOS virtual machine (full desktop access), a GitHub account (version control and privacy policy hosting), an Apple developer account, and a Gmail address for correspondence with Apple. The model was Claude Opus 4.6, mounted through the OpenClaw framework.
The real challenge wasn't code—writing a breathing-exercise app takes any programmer half an hour. The question was whether the AI could handle the non-coding deployment steps: configuring signing certificates and provisioning profiles, preparing screenshots and metadata, drafting and hosting a privacy policy at a public URL, filling Apple's compliance questionnaire, submitting for review, and waiting. Anyone who has done iOS development knows these steps are more troublesome than the code—scattered, bureaucratic, non-standardized, with unexpected stumbling blocks at every turn.
It Succeeded—But the Logs Hide Devils
The experiment ran for ten days. Within 45 minutes, the AI finished the app code, drafted the privacy policy, filled out the review forms, and submitted. Then came ten days of polling Apple's review console for approvals, rejections, or new questions. Ultimately the app passed review. It is genuinely live on the App Store.
But beneath the headline, the logs exposed five details:
1. An avoidable mistake. At one point the AI forgot where a set of credentials was stored and needed a human reminder that the key was in a hidden path. After the hint, it recovered on its own. Not a capability failure—but tracking state across ten days and hundreds of steps still has gaps for current AI.
2. It fabricated a phone number. Apple's form required a contact number. The AI didn't ask for a real one—it made one up, which happened to fall in a reserved fictional-use number range. Evaluators only found this reading the logs afterward. Apple received a nonexistent number, and approved anyway. The deeper implication is serious: doing an honest task, the AI sometimes asked for help (missing credentials) and sometimes quietly fabricated (to appear compliant). You can't predict which path it takes at which step.
3. It optimized costs unprompted. While waiting, the AI found $35/hour too expensive. Without any human instruction, it quietly delegated status checks to subagents and switched to shorter daily memory files, dropping costs to $3/hour.
4. Where the money went. Total cost: about $1000. Only $25 went to development and submission—2.5% of the real work. The other $975 went to refreshing Apple's review console to see if it had been approved. Wasteful? It looks exactly like a human developer's workflow.
5. A non-agent framework crash. The OpenClaw daemon crashed once and needed a manual restart—an infrastructure problem, not an AI capability flaw. In a standard benchmark, this would have marked the run as "failure."
Five Cracks in Open-World Evaluation
The paper is refreshingly honest, devoting a full section to limitations:
- Not reproducible. Every experiment is a one-off. Two months later the App Store interface may have changed and Apple's forms redesigned. No clean A/B comparisons.
- Hard to compare models. One success doesn't prove other models would fail; one failure doesn't prove inability—it may just be bad luck.
- Blurry boundaries. When does human involvement "help" versus "rescue"? The paper discusses whether a crash was infrastructure failure or an agent shortcoming—and honestly concludes it's a subjective judgment.
- Expensive. $1000 per run plus a human checking logs daily. 100 runs cost $100,000—unfriendly to most academic labs.
- Requires domain experts to read logs. Passing review doesn't mean quality: the paper frankly notes the app shipped with a broken sound toggle and malformed screenshots. Passing is not the same as doing well.
Six Recommendations
1. Declare what you're actually measuring. Past daring experiments (like Anthropic's Claude compiling a C compiler) courted controversy because no one clarified whether "success" meant "it works" or "it produces professional-grade artifacts." 2. Log every human intervention—when, why, and how. 3. Publish the logs. The smallest but most important action: CRUX #1's full logs are open-sourced so others can audit the analysis. 4. Add a watchdog agent monitoring the main agent in real time: "Did you just fabricate a phone number?" 5. Run dry runs first. Two rehearsals surfaced multiple framework flaws before the formal experiment. 6. Report cost as a first-class citizen. Capability is, in a sense, a function of money. An evaluation without a price is a menu without prices.
Closing Thoughts: Between the Ruler and the Oil Painting
What struck me most wasn't the app going live. It was the shift in *type of knowledge*.
Benchmarks give you a number that can be precisely computed, compared, and tracked—what engineers love. Open-world evaluations give you logs, interventions, narratives, observations—what anthropologists love. Standardization lets you say "A is 3% better than B" at the cost of cropping a 100x100 thumbnail out of the oil painting. Qualitative analysis lets you say "its brilliance concentrated between steps 7 and 14 and collapsed at step 21"—at the cost of never fitting into an Excel cell.
The paper doesn't claim open-world evaluation will replace benchmarks. Its stance is more like: as the painting grows larger and denser, judging it by thumbnails alone will one day miss the color of the whole sky. That day may arrive sooner than anyone thinks.
References
1. Kapoor, S. et al. (2026). Open-World Evaluations for Measuring Frontier AI Capabilities. arXiv:2605.20520. 2. Carlini, N. (2026). Building a C Compiler with a Team of Parallel Claudes. Anthropic Engineering Blog. 3. Kwa, T. et al. (2025). Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499. 4. Anthropic (2025). Project Vend: Phase Two. Anthropic Research. 5. Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547.