ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents

Forum topic · 小凯 · 2026-05-14

Summary

ToolCUA is an end-to-end Computer Use Agent (CUA) framework that learns to optimally alternate between atomic GUI actions (click, type) and high-level tool calls (e.g., API-based file operations). Hybrid action spaces often confuse CUAs about when to continue with GUI actions versus switch to tools, yielding suboptimal execution paths. The paper addresses three obstacles: scarcity of high-quality interleaved GUI-Tool trajectories, the cost and fragility of collecting real tool trajectories, and the lack of trajectory-level supervision for path selection. The authors introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes static GUI trajectories with synthesized grounded tool libraries, avoiding manual engineering; a tool-guided GUI RFT stage combining warmup SFT and single-turn RL to improve decisions at key GUI-Tool switching points; and online Agentic RL in high-fidelity GUI-Tool environments guided by a tool-efficient path reward. On OSWorld-MCP, ToolCUA achieves 46.85% accuracy, a roughly 66% relative improvement over baselines and a new SOTA among comparable-scale models, plus a 3.9% gain over the pure-GUI setting, demonstrating effective GUI-Tool orchestration.

Paper Overview

Research Area: ML Authors: Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye Published: 2026-05-12 arXiv: 2605.12481

Abstract

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection.

The authors propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm:

1. Interleaved GUI-Tool Trajectory Scaling Pipeline — repurposes abundant static GUI trajectories and synthesizes grounded tool libraries, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. 2. Tool-guided GUI RFT — combines warmup SFT with single-turn RL to improve decision-making at key GUI-Tool switching points. 3. Online Agentic RL — optimizes ToolCUA in high-fidelity GUI-Tool environments, guided by a tool-efficient path reward that encourages appropriate tool usage and shorter execution paths.

Results

On OSWorld-MCP, ToolCUA achieves 46.85% accuracy, a relative improvement of about 66% over baselines, establishing a new SOTA among comparable-scale models.
Compared with the pure-GUI setting, it gains an additional 3.9%, demonstrating effective GUI-Tool orchestration.

--- *Auto-collected on 2026-05-14*