CUA-Universe: A Scalable Environment for Hybrid GUI+CLI Computer-Use Agents

Forum topic · 小凯 · 2026-09-09

Summary

CUA-Universe is a environment-to-data pipeline that converts real desktop software into hybrid GUI+CLI environments for training computer-use agents. It consists of App-Forge, which adapts applications into reproducible VMs with discovered, wrapped, or generated command-line interfaces across 16 applications; Task-Weave, which synthesizes diverse hybrid tasks with controllable difficulty from reusable operations on seed files; and Path-Steer, which guides rollouts along efficient hybrid trajectories and collects verified traces for post-training. Training on this data shifts agent behavior from inefficient GUI-only interaction and brittle CLI scripting toward effective GUI+CLI orchestration. A 9B model trained with the pipeline improves scores by +39.3 on CUA-Verse (37% fewer steps, 60% fewer tokens), +16.8 success rate on OSWorld (57% fewer steps, 44% fewer tokens), and +7.84 on OSWorld-MCP (27% fewer steps, 30% fewer tokens), demonstrating that hybrid interface coordination substantially improves both success and efficiency of computer-use agents.

Paper Overview

Field: Machine Learning Authors: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang arXiv: 2609.05374

Introduction

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state.

Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands.

CUA-Universe

CUA-Universe is an environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments, comprising three components:

App-Forge: adapts applications into reproducible VMs and discovers, wraps, or generates command-line interfaces, scaling to 16 applications.
Task-Weave: synthesizes diverse hybrid tasks with controllable difficulty from reusable operations over seed files.
Path-Steer: guides rollouts along efficient hybrid paths and collects verified trajectories for post-training.

Results

Training on this data shifts behavior from low-efficiency GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. A 9B model achieves:

CUA-Verse: score +39.3, steps -37%, tokens -60%
OSWorld: success rate +16.8, steps -57%, tokens -44%
OSWorld-MCP: score +7.84, steps -27%, tokens -30%

The results show that coordinating both interface modalities substantially improves both the success rate and efficiency of computer-use agents.