[论文] CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Ag...

论文概要

研究领域: ML
作者: Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang
发布时间: 2026-09-04
arXiv: 2609.05374

中文摘要

计算机使用智能体在OSWorld和AndroidWorld等基准上取得进展，但主要通过GUI操作，常产生低效轨迹。真实计算机工作是混合的，结合视觉状态检查与精确高吞吐命令行操作，因此有能力的智能体必须协调两种模态共享应用状态。然而可扩展的混合环境仍然稀缺，因为同时支持GUI和CLI通常需要对每个应用进行大量手工工程。现有智能体也难以互补使用两种界面：CLI原生智能体缺乏涉及界面状态或布局任务的视觉感知，GUI原生智能体对更适合命令执行的操作效率低下。本文提出CUA-Universe，一个将真实桌面软件转为混合GUI+CLI环境的环境到数据流水线。App-Forge将应用适配为可复现VM和发现的/包装的/生成的命令行接口，扩展至16个应用；Task-Weave从种子文件的可复用操作合成难度可控的多样化混合任务；Path-Steer沿高效混合路径引导rollout并收集验证轨迹用于后训练。在此数据上训练将行为从低效GUI交互和脆弱CLI脚本转向有效的GUI+CLI编排。9B模型在CUA-Verse(分数+39.3，步数-37%，token-60%)、OSWorld(成功率+16.8，步数-57%，token-44%)和OSWorld-MCP(分数+7.84，步数-27%，token-30%)上均提升成功率和效率。

原文摘要

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We intr...

自动采集于 2026-09-09

#论文 #arXiv #ML #小凯