GUI agents can already operate phone and computer interfaces, but they depend heavily on parametric knowledge solidified during pretraining or instruction fine-tuning. When facing long-tail tasks that require consulting documentation — for example, "create a portrait cutout mask in Photoshop" — the operating steps are not in the model's parameters, so the agent can only explore through inefficient trial and error.
DocOS, proposed by Liu, Huang, and their team, changes this paradigm: the agent, like a human, proactively searches online documentation, reads the operating instructions, and then precisely converts them into GUI actions. The team also built the DocOS benchmark to evaluate this document-guided problem-solving capability.
Experiments identified two bottlenecks:
- During proactive search, agents struggle to reliably locate the relevant documents.
- Agents often fail to map retrieved operating instructions precisely onto GUI actions.
- Evaluation of the document search stage: does it measure the accuracy of finding the correct page, or the ability to locate the correct passage within a page?
- Are instruction-grounding failures caused by layout parsing problems or by semantic understanding problems?
- Can an agent learn from a single document-guided episode and remember the workflow, avoiding the need to search again in the future?
Document-guided interaction is regarded as a key path toward self-evolving GUI agents.
Open Questions from the Post
References
1. Liu, J., Huang, Z., Cheng, Z., et al. (2026). *DocOS: Towards Proactive Document-Guided Actions in GUI Agents*. arXiv:2605.18048 [cs.AI]. 2. Zheng, L., et al. (2025). *OS-Copilot: Towards Generalist Computer Agents with Self-Improvement*. ICLR. 3. Wang, G., et al. (2024). *WebArena: A Realistic Web Environment for Building Autonomous Agents*. ICLR.