Overview
Field: Machine Learning Authors: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang arXiv: 2609.05374
Motivation
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but they still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid — it combines visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state.
Two key problems:
- Scalable hybrid environments are scarce: supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application.
- Existing agents struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands.
The CUA-Universe Pipeline
CUA-Universe is an environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments:
1. App-Forge — adapts applications into reproducible VMs and provides discovered, wrapped, or generated command-line interfaces, scaling to 16 applications. 2. Task-Weave — synthesizes diverse hybrid tasks with controllable difficulty from reusable operations on seed files. 3. Path-Steer — guides rollouts along efficient hybrid paths and collects verified trajectories for post-training.
Results
Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. A 9B model improves success rate and efficiency across three benchmarks:
| Benchmark | Score/Success | Steps | Tokens | |---|---|---|---| | CUA-Verse | score +39.3 | -37% | -60% | | OSWorld | success +16.8 | -57% | -44% | | OSWorld-MCP | score +7.84 | -27% | -30% |
--- *Auto-collected 2026-09-09*