Paper Overview
Research Area: Computer Vision (CV)
Authors: Miaosen Zhang, Xiaohan Zhao, Zhihong Tan, Zhou Huoshen, Yijia Fan, Yifan Yang, Kai Qiu, Bei Liu, Justin Wagle, Chenzhong Yin, Mingxi Cheng, Ji Li, Qi Dai, Chong Luo, Xu Yang, Xin Geng, Baining Guo
Release Date: 2026-05-12
arXiv: 2605.12501
Summary
Computer-use agents (CUAs) automate on-screen work, as illustrated by systems like GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions remains poor, limiting user trust.
The authors analyze failure cases from advanced models and identify a long-tail pattern in GUI operations: a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. They hypothesize that this problem largely stems from the scarcity of training data covering complex interactions.
To tackle this, the paper makes two main contributions:
1. CUActSpot benchmark: A new evaluation suite that tests model capabilities on complex interactions across five modalities—GUI, text, table, canvas, and natural image—and supports a variety of actions including click, drag, draw, and others. This coverage goes well beyond prior benchmarks that focus mainly on clicking.
2. Renderer-based synthetic data pipeline: An automated pipeline that generates scenes for each modality, records screenshots along with element coordinates, and uses a large language model to produce matching instructions and action traces. This yields a corpus tailored to the long tail of complex interactions.
After training on this synthetic corpus, the resulting model Phi-Ground-Any-4B achieves the best performance among open-source models with fewer than 32B parameters.
Key Points
- Long-tail failure mode: Advanced CUAs disproportionately fail on complex, low-frequency interactions rather than routine ones.
- Root cause hypothesis: Data scarcity for complex interactions is the primary bottleneck, not model capacity alone.
- CUActSpot benchmark: Evaluates five modalities (GUI, text, table, canvas, natural image) and diverse action types (click, drag, draw, etc.), filling a gap left by click-centric benchmarks.
- Synthetic data pipeline: Renderer-generated scenes + element coordinates + LLM-generated instructions and trajectories provide scalable supervision for rare interactions.
- Result: Phi-Ground-Any-4B is the strongest open-source model under 32B parameters when trained on the proposed corpus.
Source
*Auto-collected on 2026-05-14*
#Paper #arXiv #CV #ComputerUseAgents #Benchmark #SyntheticData