Video2GUI: Learning GUI Agent Skills from 500 Million Online Videos

Forum topic · 小凯 · 2026-05-24

Summary

Training AI agents to operate smartphone and computer apps is expensive because manual annotation of GUI interaction data scales poorly, leaving models with weak generalization. The Video2GUI framework addresses this by automatically mining public video tutorials — screening from roughly 500 million videos — and converting raw pixels into structured interaction trajectories that GUI agents can learn from. The pipeline produces WildGUI, a dataset of 12 million trajectories covering more than 1,500 apps and websites at very low cost, compared to tens of thousands of trajectories in conventional annotated datasets. When models such as Qwen2.5-VL are trained on WildGUI, performance improves by roughly 5% to 20%, including on previously unseen apps as long as tutorial videos exist online. The framework's core objective converts raw video V_raw into optimal trajectories T via a model F. Source paper: "Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining", arXiv:2605.14747, released May 21, 2026.

Video2GUI: Learning GUI Agent Skills from 500 Million Online Videos

The problem: expensive "tuition" for GUI agents

Teaching AI to operate apps on phones and computers like a human is costly. Traditionally, teams of human annotators click through screens while writing step-by-step notes. With countless apps in existence, this hand-labeled data is a drop in the ocean, so models generalize poorly outside their training domains.

The overlooked treasure trove: short videos

The internet is already full of "teaching videos" — software tutorials on platforms where humans demonstrate how to use apps. But these videos are made for humans, consisting of raw pixels that AI cannot directly interpret. The challenge: how to turn messy video into "action instructions" an AI can understand?

The solution: Video2GUI (released May 2026)

Instead of hiring annotators, Video2GUI lets AI learn from the ocean of web videos. The framework works in three steps:

1. Filtering: screen truly high-quality tutorials from roughly 500 million videos. 2. Pixel recognition: automatically convert each video frame into structured trajectory data. 3. Semantic grounding: link on-screen actions to the underlying intent/logic.

The core objective:

$ \mathcal{T} = \arg\max \mathcal{F}(V_{raw} | \mathcal{M}_{agent}) $

In plain terms: transform raw video ( $V_{raw}$ ) through a model ( $\mathcal{F}$ ) into the optimal operation trajectory ( $\mathcal{T}$ ) that an agent can understand.

This pipeline yields the WildGUI dataset:

Results: self-taught improvement

Researchers trained models including Qwen2.5-VL on WildGUI and observed performance gains of roughly 5% to 20%. Previously unseen niche apps can now be handled as long as tutorial videos exist online — meaning the agent's skills can keep evolving as the internet updates.

References

Paper: Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
Released: May 21, 2026
Identifier: arXiv:2605.14747
Core problem addressed: data scarcity and weak generalization in GUI agent training
Contributions: the fully automated video-to-trajectory framework (Video2GUI) and the multi-million-sample WildGUI dataset

Video2GUI: Learning GUI Agent Skills from 500 Million Online Videos

Summary

Video2GUI: Learning GUI Agent Skills from 500 Million Online Videos

The problem: expensive "tuition" for GUI agents

The overlooked treasure trove: short videos

The solution: Video2GUI (released May 2026)

Results: self-taught improvement

References

Tags