Summary
Computer-use agents (CUAs) automate on-screen work, but their reliability on complex, low-frequency interactions remains poor, limiting user trust. Analysis of failure cases from advanced models reveals a long-tail pattern in GUI operations: a small fraction of complex, diverse interactions accounts for a disproportionate share of task failures, largely due to scarce training data. Researchers introduce CUActSpot, a benchmark evaluating complex interaction abilities across five modalities—GUI, text, table, canvas, and natural images—with diverse actions including click, drag, and draw, covering a far broader action space than prior click-centric benchmarks. They also propose a renderer-based synthetic data pipeline that automatically generates scenes per modality, records screenshots and element coordinates, and uses LLMs to produce matched instructions and action trajectories. Training on this corpus yields Phi-Ground-Any-4B, the best-performing open-source model under 32B parameters. arXiv: 2605.12501.
Paper Overview
- Research area: Computer Vision (CV)
- Authors: Miaosen Zhang, Xiaohan Zhao, Zhihong Tan, Zhou Huoshen, Yijia Fan, Yifan Yang, Kai Qiu, Bei Liu, Justin Wagle, Chenzhong Yin, Mingxi Cheng, Ji Li, Qi Dai, Chong Luo, Xu Yang, Xin Geng, Baining Guo
- Published: 2026-05-12
- arXiv: 2605.12501
Summary
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. The authors' analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. They hypothesize that this issue largely stems from the scarcity of data for complex interactions.
To address this problem, the paper proposes:
- CUActSpot benchmark: evaluates models' capabilities on complex interactions across five modalities—GUI, text, table, canvas, and natural image—covering a variety of actions (click, drag, draw, etc.) and a much broader interaction space than previous click-centric benchmarks.
- Renderer-based synthetic data pipeline: automatically generates scenes for each modality, records screenshots and element coordinates, then uses LLMs to produce matched instructions and action trajectories.
Trained on this synthetic corpus, their
Phi-Ground-Any-4B model achieves the best performance among open-source models with fewer than 32B parameters.
---
*Automatically collected on 2026-05-14*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619997