ToolCUA: Learning Optimal GUI-Tool Path Orchestration for Computer Use Agents

Forum topic · 小凯 · 2026-05-14

Summary

This paper introduces ToolCUA, an end-to-end computer use agent (CUA) that learns when to continue with atomic GUI actions versus switch to high-level tool calls such as API-based file operations. Existing CUAs struggle with hybrid action spaces because high-quality interleaved GUI-Tool trajectories are scarce, real tool trajectories are costly and brittle to collect, and trajectory-level supervision for GUI-Tool path selection is lacking. ToolCUA addresses these gaps through a staged training paradigm: an Interleaved GUI-Tool Trajectory Scaling Pipeline reuses abundant static GUI trajectories to synthesize grounded tool libraries without manual engineering, followed by Tool-Guided GUI RFT combining warm-up SFT with single-turn RL to refine decisions at critical GUI-Tool switching points. A high-fidelity GUI-Tool environment with online agentic RL guided by tool-efficient path rewards further optimizes execution. On OSWorld-MCP, ToolCUA achieves 46.85% accuracy, a roughly 66% relative improvement over baselines and a new state-of-the-art among comparable-scale models, with a 3.9% gain over pure GUI settings.

Paper Overview

Field: Machine Learning
Authors: Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye
Released: 2026-05-12
arXiv: 2605.12481

Summary

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations. However, this hybrid action space often leaves them uncertain about when to continue with GUI actions versus switch to tools, leading to suboptimal execution paths. The difficulty stems from:

Scarcity of high-quality interleaved GUI-Tool trajectories
The cost and brittleness of collecting real tool trajectories
Lack of trajectory-level supervision for GUI-Tool path selection

This paper proposes ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm.

Method

1. Interleaved GUI-Tool Trajectory Scaling Pipeline — repurposes abundant static GUI trajectories and synthesizes grounded tool libraries, enabling diverse GUI-Tool trajectories without manual engineering or real tool trajectory collection. 2. Tool-Guided GUI RFT — combines warm-up SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. 3. Online Agentic RL in a High-Fidelity GUI-Tool Environment — optimizes ToolCUA further, guided by tool-efficient path rewards that encourage appropriate tool usage and shorter execution paths.

Results

On OSWorld-MCP, ToolCUA achieves 46.85% accuracy, about a 66% relative improvement over baselines.
Establishes a new state-of-the-art among comparable-scale models.
A 3.9% improvement over the pure GUI setting demonstrates the effectiveness of GUI-Tool orchestration.

Key Points

Hybrid GUI + tool action spaces require explicit orchestration; current agents lack trajectory-level supervision for switching decisions.
A synthetic data pipeline can replace costly real tool trajectory collection by augmenting static GUI trajectories with grounded tool libraries.
Staged training (SFT → single-turn RL → online agentic RL with tool-efficiency rewards) is effective for teaching optimal GUI-Tool path selection.
ToolCUA delivers substantial accuracy gains on OSWorld-MCP and validates that mixing GUI actions with tool calls outperforms GUI-only execution.

ToolCUA: Learning Optimal GUI-Tool Path Orchestration for Computer Use Agents

Summary

Paper Overview

Summary

Method

Results

Key Points

Tags