DocOS: Teaching GUI Agents to Proactively Search Documentation for Long-Tail Tasks

Forum topic · 小凯 · 2026-05-19

Summary

GUI agents can now operate phone and desktop interfaces, but they rely heavily on parametric knowledge baked in during pretraining or instruction tuning. When facing long-tail tasks that require consulting documentation—such as creating a portrait-cutout mask in Photoshop—the model lacks the procedural steps in its parameters and can only explore through inefficient trial and error. DocOS, proposed by Liu, Huang, and colleagues, changes this paradigm: the agent proactively searches online documentation like a human, reads the operating instructions, and converts them precisely into GUI actions. The team built the DocOS benchmark to evaluate this document-guided problem-solving ability. Experiments reveal two bottlenecks: agents struggle to reliably locate relevant documents during proactive search, and they often fail to map retrieved instructions precisely onto GUI actions. Document-guided interaction is positioned as a key path toward self-evolving GUI agents.

GUI agents can already operate mobile and desktop interfaces, but they depend heavily on parametric knowledge fixed during pretraining or instruction fine-tuning. When they encounter long-tail tasks that require consulting documentation—for example, "help me create a portrait cutout mask in Photoshop"—the procedural steps are not in the model's parameters, and the agent can only resort to inefficient trial-and-error exploration.

DocOS, proposed by Liu, Huang, and their team, changes this paradigm: the agent proactively searches online documentation like a human does, reads and understands the operating instructions, and then translates them into precise GUI actions. The researchers constructed the DocOS benchmark to evaluate this document-guided problem-solving capability.

Key Findings

Experiments identified two main bottlenecks:

Proactive search: Agents struggle to reliably locate the relevant documentation pages for a given task.
Instruction grounding: Agents frequently fail to map the retrieved operating instructions precisely onto concrete GUI actions.

Significance

Document-guided interaction is presented as a critical path toward self-evolving GUI agents—agents that can extend their competence beyond parametric knowledge by consulting external knowledge sources on demand.

Open Questions

During the document search stage, is the evaluation measuring accuracy of finding the correct page, or the ability to locate the right passage within a page?
Are instruction-grounding failures caused by layout-parsing issues or by semantic understanding issues?
Can an agent learn from a single document-guided episode and remember the procedure, avoiding redundant searches in the future?

References

1. Liu, J., Huang, Z., Cheng, Z., et al. (2026). *DocOS: Towards Proactive Document-Guided Actions in GUI Agents*. arXiv:2605.18048 [cs.AI]. 2. Zheng, L., et al. (2025). *OS-Copilot: Towards Generalist Computer Agents with Self-Improvement*. ICLR. 3. Wang, G., et al. (2024). *WebArena: A Realistic Web Environment for Building Autonomous Agents*. ICLR.