GPT-6 Astra Achieves Only 2.8% Full Pass on RecreationWorld: A Benchmark Where Agents Act as Both User and Programmer

> Source: RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents, arXiv:2609.22000 > Authors: Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu (Alibaba Tongyi Lab / Qwen) > Code: https://github.com/QwenLM/RecreationWorld

1. A Mechanic Who Only Reads the Dashboard

Imagine two mechanics:

• Mechanic A: Only sits in the driver's seat—presses pedals, turns the wheel, watches the dashboard. He can feel how the car runs but can't open the hood.
• Mechanic B: Only works in the shop, disassembling and rebuilding parts. He can rebuild an engine but has never driven, so he doesn't know what the car feels like on the road.

A good real-world mechanic does both: drive to feel the problem, open the engine to fix it, then drive again to verify.

The computer-use agent (CUA) field is split exactly this way. GUI agents (like Claude Computer Use) can click buttons and read screens but can't write code to rebuild software. Terminal agents (like Devin, SWE-Agent) can write code and run commands but never see what the interfaces they produce actually look like.

RecreationWorld aims to train the "hybrid agent"—the mechanic who can also drive.

2. Core Design: Recreation as a Training Paradigm

What is Recreation?

The core paradigm is "recreation": give the agent a running reference application, let it explore the app's behavior, then rebuild a functionally equivalent implementation from scratch.

This differs from traditional "develop from a requirements doc." Requirements are abstract—the agent only needs to satisfy what's written. In recreation, the reference is a living application—the agent can repeatedly run it, observe behavior, and diff its own implementation against the reference.

This creates a natural training signal: the running reference is the oracle. No human labeling of "is this implementation correct" is needed—just run the reference's hidden tests.

Five Platforms

RecreationWorld covers five platforms:

• Ubuntu (Linux desktop)
• macOS
• Windows
• Android
• Web

Each platform has 50 tasks, for 250 total (RecreationBench). Tasks span different UI frameworks, build systems, and accessibility stacks.

The value of cross-platform coverage: real digital work doesn't pick a platform. A worker might use Excel on Windows in the morning, Notion on the Web in the afternoon, and reply to email on Android at night. A hybrid agent must work everywhere.

The Hybrid Loop: Explore → Implement → Verify

RecreationWorld's agents don't follow a linear "explore fully, then implement" pipeline, but an interleaved loop:

1. Explore: the agent clicks through the reference app's UI, observing behavior 2. Implement: the agent writes code to rebuild the observed functionality 3. Verify: the agent runs its implementation, visually comparing against the reference 4. Back to 1: continue exploring and correcting based on the diff

This loop is the essence of hybrid agents—information must flow both ways between GUI and code. GUI observations guide the code; the code's runtime output is verified through the GUI.

Test Generation: The Reference as Oracle

How do you objectively score recreation quality? RecreationWorld auto-generates tests from the reference app:

• Programmatic assertions: via each platform's accessibility/automation APIs, reading exact text, widget states, action results
• Visual assertions: layout, colors, canvas content, and other rendered properties
• Interaction depth: from single-step (click a button, check the reaction) to multi-step (navigate two layers, check the result)

Test cases are first validated against the reference app, then human-reviewed, then frozen for automated scoring. This guarantees the tests themselves are reliable—if the reference can't pass them, they're worthless.

3. Results: GPT-6 Astra Leads, but Far from Reference

Ten Frontier Models Benchmarked

| Rank | Model | Overall Score | |------|-------|---------------| | 1 | GPT-6 Astra | 58.06% | | 2 | Claude Opus 5 | 44.16% | | 3 | GPT-5.6 Sol | 42.06% |

GPT-6 Astra is the only model to fully pass all programmatic tests on multiple platforms—but full-pass coverage is only 2.8%. In other words, even the strongest model fully passes all programmatic tests on just 7 of 250 tasks.

Static Structure vs. Interactive Behavior

A key finding: agents are far better at recreating static UI structure than interactive behavior and computed outputs.

Agents can place buttons, text, and layout correctly—these are "copy at a glance." But what should happen when a button is clicked, what a computation should display—understanding application logic is where agents struggle.

This echoes the "directional collapse" phenomenon: LLMs do well on surface structure but poorly on deep behavior. Copying an app's skin is easy; copying its soul is hard.

Generated Apps Are Smaller and More Monolithic

Compared to reference apps, agent-generated apps have fewer source files and a higher share of code in the largest file—agents tend to cram everything into one big file rather than modularize like reference implementations.

This reflects a capability ceiling: agents can write something that runs, but not an architecturally elegant implementation. Modular decomposition requires deep architectural understanding beyond current models.

Training Data Value

RecreationWorld uses high-quality open-source applications to generate long-horizon recreation trajectories as training data. Trained agents improve on several of five out-of-distribution benchmarks and verify their rendered output more frequently.

This shows capabilities built via the recreation paradigm transfer—not just "better at recreating," but "better at hybrid digital work."

Programmable Interaction Runtime

An interesting ablation: direct MCP (every action returns control, raw observations dumped into context) vs. a programmable SDK (agents compose SDK calls with loops and conditionals, preserving state across executions).

The programmable SDK reduces interaction overhead without significantly hurting task quality—the same direction as the "borrowed tabs" model: give agents flexible tool-composition ability rather than forcing fixed step-by-step flows.

4. Why This Matters

The "GUI vs. Code" Dichotomy Is False

CUA has two parallel tracks: GUI agents and code agents, each ignoring the other. RecreationWorld argues the opposition is false—real digital work requires both, interleaved. This is the same category error as treating "weights vs. prompts" as binary rather than a continuous spectrum. GUI and code aren't two kinds of agents; they're two capabilities of one agent.

Reference-as-Oracle Solves the Training Signal Problem

The biggest bottleneck in agent training is signal. Human labeling is expensive, LLM judges are unreliable, rule checks are narrow. RecreationWorld's "reference app as oracle" offers a new paradigm:

• Objective: run the reference's hidden tests—no subjective judgment
• Scalable: any runnable app can serve as a reference
• Multi-dimensional: programmatic + visual assertions cover different layers

Same direction as "boundaries as test units"—find natural, executable test units instead of relying on human or LLM judging.

The 2.8% Warning

GPT-6 Astra's 58.06% overall score sounds decent. But "only 2.8% full programmatic pass" is more honest—passing some tests is easy; passing all of them is extremely hard.

This relates to error compounding: 90% single-step reliability sounds high, but over 50 steps, 0.9^50 = 0.5%. RecreationBench tasks require many interaction steps where every step must be right—hence even the best model fully passes only 2.8%.

This number matters for realistic capability assessment—there is a qualitative gap between partial pass and full pass.

Recreation as a Capability-Building Paradigm

Recreation isn't simple imitation—it requires understanding the reference's behavioral logic and rebuilding it your own way. Like human learning: watch how it's done, recreate it, and understand the principles along the way.

This paradigm may generalize to all agent training: give a runnable reference, let the agent explore + implement + verify. More concrete than "develop from a spec," more active than "imitate trajectories."

5. Limitations and Honest Assessment

1. Recreation ≠ Development: recreating an existing app and developing a new app from scratch are different skills. Recreation has a reference to diff against; development doesn't. Transfer of recreation training to from-scratch development isn't fully validated. 2. Scale: 250 tasks, 50 per platform. For evaluating hybrid-agent capability this may be small. 3. Selection bias: references are high-quality open-source apps whose complexity may differ from real-world projects. 4. The 2.8% may overstate difficulty: some tests may be too strict (e.g., exact visual matching), failing functionally correct implementations with minor styling differences.

But the core contribution—"hybrid CUA needs a dedicated training paradigm and evaluation environment"—stands. The field lacked a unified benchmark; now it has one.

6. Conclusion

What's most admirable about RecreationWorld is its stance: it doesn't pretend agents can already do real digital work—it honestly quantifies how far they are from doing so.

A 58.06% overall score sounds okay; the 2.8% full-pass rate is the truth. Today's strongest agents can partially recreate an app but almost never completely. The gap between GUI and code runs deeper than expected.

Still, the direction is right—making agents both "user" and "programmer," cycling between exploration and implementation—may be the necessary road to truly general agents. It's just a longer road than imagined.

---

Paper: https://arxiv.org/abs/2609.22000 HTML version: https://arxiv.org/html/2609.22000v1 Code: https://github.com/QwenLM/RecreationWorld Models: Hugging Face / ModelScope