Video2GUI: Teaching AI Agents to Use Apps by Mining 500 Million Web Videos

Forum topic · 小凯 · 2026-05-24

Summary

Training GUI agents to operate mobile and desktop apps has long been limited by expensive, small-scale human annotation data. The Video2GUI framework, released May 21, 2026 (arXiv:2605.14747), tackles this bottleneck by automatically converting internet tutorial videos into usable training data. The pipeline filters high-quality tutorials from roughly 500 million videos, converts video frames into structured interaction trajectories, and links visual actions with underlying intent. The result is WildGUI, a dataset of 12 million trajectories spanning more than 1,500 apps and websites, produced at very low cost through a fully automated pipeline. Fine-tuning models such as Qwen2.5-VL on WildGUI improved performance by 5% to 20%, including on apps never seen during training. Because it learns from continuously updated web video, the approach offers a scalable path to more generalizable GUI agents.

Video2GUI: When AI Learns to Use Apps from 500 Million Videos

The Problem: Expensive Training Data

Training AI agents to operate apps on phones and computers has traditionally required large teams of human annotators who click through screens while writing step-by-step descriptions. With countless apps in existence, manually labeled data covers only a tiny fraction, leaving models with poor generalization to unfamiliar software.

The Untapped Resource: Tutorial Videos

The internet is full of software tutorial videos — people demonstrating how to use apps on platforms like Bilibili and Douyin. But these videos are made for humans: raw pixels that AI cannot directly interpret as actions. The core challenge is converting messy video content into structured action instructions a model can learn from.

The Solution: Video2GUI

Released on May 21, 2026, the Video2GUI framework automates the conversion of web videos into GUI agent training data via three stages:

1. Filtering: selecting genuinely high-quality tutorials from roughly 500 million videos. 2. Perception: automatically converting each video frame into structured trajectory data. 3. Reasoning: connecting on-screen actions with the logic behind them.

The core objective is formalized as:

$\mathcal{T} = \arg\max \mathcal{F}(V_{raw} \mid \mathcal{M}_{agent})$

That is, transforming raw video ( $V_{raw}$ ) through a model ( $\mathcal{F}$ ) into optimal, machine-readable interaction trajectories ( $\mathcal{T}$ ).

The WildGUI Dataset

Results

When models such as Qwen2.5-VL were fine-tuned on WildGUI, benchmark performance improved by 5% to 20%. Notably, agents gained the ability to operate unfamiliar apps — as long as tutorial videos exist online. Since the internet keeps producing new video content, this offers a self-refreshing, ever-current source of training signal for GUI agents.

References

Paper: Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
Released: May 21, 2026
arXiv: arXiv:2605.14747
Contributions: Fully automated video-to-trajectory framework (Video2GUI) and the million-scale WildGUI dataset, addressing data scarcity and weak generalization in GUI agents.