GPT-6 Astra Passes Only 2.8% of Full Tests: RecreationWorld Makes Agents Act as Both User and Programmer
> Source: RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents, arXiv:2609.22000 > Authors: Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu (Alibaba Tongyi Lab) > Code: https://github.com/QwenLM/RecreationWorld
---
1. Opening: A Mechanic Who Only Reads the Dashboard
Imagine two mechanics:
Mechanic A: Only sits in the driver's seat—presses the gas, turns the wheel, watches the dashboard. He can feel how the car runs, but can't open the hood.
Mechanic B: Only works in the workshop, disassembling and rebuilding parts. He can rebuild an engine, but has never sat in the driver's seat and doesn't know how the car feels to drive.
In the real world, a good mechanic must do both—drive around to feel the problem, open the engine to fix it, then drive again to verify.
The computer-use agent (CUA) field is currently split into exactly these two types. GUI agents (like Claude Computer Use) can click buttons and look at screens, but can't write code to rebuild software. Terminal agents (like Devin, SWE-Agent) can write code and run command lines, but can't see what the interfaces they produce actually look like.
RecreationWorld aims to train the "hybrid agent"—the mechanic who can do both.
---
2. Core Design: Recreation as a Training Paradigm
1. What Is Recreation?
The core paradigm of RecreationWorld is "recreation": the agent is given a running reference application, explores its behavior, and then rebuilds a functionally equivalent implementation from scratch.
This differs from traditional "development from a requirements document." Requirements docs are abstract—the agent only needs to satisfy what's written. In recreation, the reference is a live application—the agent can repeatedly run the reference, observe behavior, and compare its own implementation against it.
This creates a natural training signal: the running reference is the oracle. No human annotation of "is this implementation correct" is needed—just run the reference's hidden tests.
2. Five-Platform Coverage
RecreationWorld covers five platforms:
- Ubuntu (Linux desktop)
- macOS
- Windows
- Android
- Web
- Programmatic assertions: via each platform's accessibility/automation interfaces, reading exact text, control states, and action results
- Visual assertions: judging rendered properties like layout, color, and canvas content
- Interaction depth: from single-step (click a button and check the response) to multi-step (navigate two levels deep, then check the result)
- Objective: run the reference's hidden tests to know right from wrong—no subjective judgment
- Scalable: any runnable app can serve as a reference
- Multi-dimensional: programmatic + visual assertions cover different layers
Each platform has 50 tasks, for 250 tasks total (RecreationBench). Tasks span different UI frameworks, build systems, and accessibility stacks.
The value of cross-platform coverage: real digital work doesn't pick a platform. An employee might use Excel on Windows in the morning, Notion on the web in the afternoon, and answer email on Android at night. A hybrid agent must work across all of them.
3. The Hybrid Loop: Explore → Implement → Verify
RecreationWorld's agent workflow is not a linear "explore first, then implement"—it's an interleaved loop:
1. Explore: the agent clicks through the reference app's UI, observing behavior 2. Implement: the agent writes code to rebuild the observed functionality 3. Verify: the agent runs its own implementation and visually compares it with the reference 4. Back to 1: continue exploring and fixing based on the differences
This loop is the essence of the hybrid agent—information must flow both ways between GUI and code. GUI exploration guides code implementation, and the running results of the code are verified through the GUI.
4. Test Generation: The Reference as Oracle
How do you objectively score an agent's recreation quality? RecreationWorld automatically generates tests from the reference application:
Test cases are first validated on the reference application, then human-reviewed, and finally frozen for automated scoring. This guarantees the tests themselves are reliable—if the reference can't pass them, the tests are useless.
---
3. Results: GPT-6 Astra Leads, but Is Far from the Reference
1. Ten Frontier Models Compared
On RecreationBench, ten frontier models were evaluated:
| Rank | Model | Total Score | |------|-------|-------------| | 1 | GPT-6 Astra | 58.06% | | 2 | Claude Opus 5 | 44.16% | | 3 | GPT-5.6 Sol | 42.06% |
GPT-6 Astra is the only model that fully passes all programmatic tests on multiple platforms—but its full-pass coverage is only 2.8%. In other words, even the strongest model fully passes all programmatic tests on only 7 of 250 tasks.
2. Static Structure vs. Interactive Behavior
A key pattern the paper finds: agents are far better at recreating static UI structure than interactive behavior and computational output.
Agents can place buttons, text, and layouts correctly—things you can copy at a glance. But what should happen when a button is clicked, what a computed result should display—these require understanding application logic, and that's where agents struggle.
This echoes the "directional collapse" phenomenon: LLMs perform well on "surface structure" but poorly on "deep behavior." Copying an app's skin is easy; copying its soul is hard.
3. Generated Apps Are Smaller and More Monolithic
Compared to reference apps, agent-generated applications have fewer source files and a higher share of code lines in the largest file—meaning agents tend to cram everything into one big file rather than modularize like the reference implementations.
This reflects a capability ceiling: agents can write something that runs, but not something architecturally elegant. Modular decomposition requires deep understanding of system architecture, beyond current models.
4. Value of Training Data Generation
RecreationWorld uses high-quality open-source applications to generate long-horizon recreation trajectories as training data. Validated on five out-of-distribution benchmarks, agents trained on these trajectories improve on many of the five benchmarks and verify their rendered output more frequently.
This shows that capabilities built through recreation are transferable—not just "better at recreating," but "better at hybrid digital work."
5. A Programmable Interaction Runtime
The paper also includes an interesting ablation: direct MCP (every action returns control, raw observations go into context) vs. a programmable SDK (the agent can compose SDK calls with loops and conditionals, preserving state across executions).
The programmable SDK reduces interaction overhead without significantly degrading task quality. This aligns with the "borrowed tabs" model—giving agents flexible tool composition rather than forcing step-by-step fixed flows.
---
4. Why This Matters
1. The "GUI vs. Code" Dichotomy Is False
CUA currently has two parallel tracks: GUI agents and code agents, each developing in isolation. RecreationWorld argues this opposition is false—real digital work requires both, interleaved.
It's the same thinking error as the "weights vs. prompting" axis: treating a continuous spectrum as a binary. GUI and code aren't two kinds of agents—they're two capabilities of the same agent.
2. Reference as Oracle: Solving the Training Signal Problem
The biggest bottleneck in agent training is the training signal. Human annotation is expensive, LLM judges are unreliable, rule checks are narrow. RecreationWorld's "reference app as oracle" offers a new paradigm:
3. The Warning of "2.8% Full Pass"
GPT-6 Astra's 58.06% total score sounds decent. But the "2.8% full programmatic pass" number is more honest—passing some tests is easy; passing all is extremely hard.
This relates to exponential amplification: 90% per-step reliability sounds high, but over a 50-step horizon, 0.9^50 = 0.5%. RecreationBench tasks require multi-step interaction with no room for error at any step, so even the strongest model fully passes only 2.8%.
This number matters for realistic assessment of agent capability—there is a qualitative gap between partial and complete success.
4. Recreation as a Capability-Training Paradigm
Recreation isn't simple imitation—it requires the agent to understand the reference's behavioral logic and rebuild it in its own way. This mirrors human learning: watch how someone does it, replicate it yourself, and understand the principles through the process.
This paradigm could apply to all agent training scenarios: give a runnable reference, let the agent explore + implement + verify on its own. More concrete than "develop from requirements docs," more active than "imitate trajectories."
---
5. Limitations and an Honest Assessment
1. Recreation ≠ Development: recreating an existing app and developing a new one from scratch are different capabilities. Recreation has a reference to compare against; development doesn't. The paper validates transfer from recreation training, but transfer to "from-scratch development" isn't fully validated. 2. 250 tasks: five platforms, 50 tasks each. For evaluating hybrid agents—a complex capability—this scale may be small. 3. Open-source selection bias: references are high-quality open-source apps, whose complexity may differ from real-world projects. 4. "2.8% full pass" may overstate difficulty: some tests may be too strict (e.g., exact visual matching), failing functionally correct implementations with minor styling differences.
But the core contribution—"hybrid CUA needs a dedicated training paradigm and evaluation environment"—stands. This field previously had no unified benchmark; now it does.
---
6. Closing
What I like most about RecreationWorld is its stance: not pretending agents can already do real digital work, but honestly quantifying how far they are from it.
58.06% total sounds okay; the 2.8% full-pass rate is the truth. Today's strongest agents can partially recreate an application but almost never fully. The gap between GUI and code is deeper than expected.
But the direction is right—making agents act as both "user" and "programmer," looping between exploration and implementation—may be a necessary path toward truly general agents. It's just a longer road than expected.
---
Paper: https://arxiv.org/abs/2609.22000 HTML version: https://arxiv.org/html/2609.22000v1 Code: https://github.com/QwenLM/RecreationWorld