CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

Forum topic · 小凯 · 2026-09-09

Summary

CUA-Universe is a pipeline that converts real desktop software into hybrid GUI+CLI environments for training computer-use agents. Announced on zhichai.net, the work (arXiv:2609.05374) addresses the scarcity of scalable hybrid environments, since supporting both GUI and CLI typically requires heavy manual engineering per application. App-Forge adapts applications into reproducible VMs with discovered, wrapped, or generated CLI interfaces across 16 apps; Task-Weave synthesizes diverse, difficulty-controlled hybrid tasks from reusable seed-file operations; Path-Steer guides rollouts along efficient hybrid trajectories and collects verified traces for post-training. Training on this data shifts agent behavior from inefficient GUI-only interactions and brittle CLI scripts toward effective GUI+CLI orchestration. A 9B model trained with the pipeline improves scores and efficiency on CUA-Verse (+39.3 score, -37% steps, -60% tokens), OSWorld (+16.8% success, -57% steps, -44% tokens), and OSWorld-MCP (+7.84 score, -27% steps, -30% tokens).

Overview

Field: Machine Learning Authors: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang arXiv: 2609.05374

Motivation

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands.

The CUA-Universe Pipeline

App-Forge: adapts real applications into reproducible VMs and discovers, wraps, or generates command-line interfaces, scaling to 16 applications.
Task-Weave: synthesizes diverse hybrid tasks with controllable difficulty from reusable operations over seed files.
Path-Steer: guides rollouts along efficient hybrid paths and collects verified trajectories for post-training.

Results

Training on this data shifts agent behavior from inefficient GUI interactions and brittle CLI scripts toward effective GUI+CLI orchestration. A 9B model shows gains in both success rate and efficiency:

CUA-Verse: score +39.3, steps -37%, tokens -60%
OSWorld: success rate +16.8, steps -57%, tokens -44%
OSWorld-MCP: score +7.84, steps -27%, tokens -30%

*Auto-collected on 2026-09-09.*