> When AI clicks the wrong button on a GUI or miscounts objects, the problem may not be model capability but how skills are stored — plain text simply cannot hold visual information.
The Blind Spot of Text Skills
Existing agents learn skills essentially by taking notes: operation steps written as text instructions, reasoning recorded as text traces, successful experiences summarized as text. This works for text-heavy tasks but fails on vision-intensive ones. Typical failure scenarios:
- GUI grounding: a button described as "the second icon in the top-right corner" breaks under different resolutions or themes
- Dense object counting: for 47 birds in one image, text can only say "count every bird" — no spatial evidence to avoid double-counting or omissions
- Visual state tracking: "the page refreshed" encodes nothing about which elements changed and where
- Declarative text logic: semantic reasoning and flow control (the traditional step 1, 2, 3)
- Reusable visual priors / source references: the new core — it doesn't describe visual information, it directly preserves visual evidence (screenshots, page regions, bounding boxes, state snapshots) as loadable attachments
- Multimodal binding protocol: specifies how text steps and visual assets are jointly retrieved and executed, e.g. step 3 "click submit" points to the button's bounding box rather than a textual guess
- GUI grounding: consistent accuracy gains over text skills — instead of trusting "click the top-right settings icon", the model relocates the icon by visual similarity from its stored bounding box
- Dense object counting: saving visual markers of "already-counted regions" prevents double counting and omissions
- Code skills: also store IDE locations, file structure, debugger variable-state screenshots
- Documentation skills: also store paragraph screenshots, table regions, chart positions
- Multimodal skills: audio waveforms, video keyframes, 3D model viewpoints
- Visual storage cost: screenshots and region markers take far more space than text
- Robustness of visual matching: theme changes, resolutions, and dynamic content degrade prior matching
- Skill generalization: visual priors trained on one website may fail on structurally different sites
- Paper: Agent Skills Should Go Beyond Text: The Case for Visual Skills
- arXiv: https://arxiv.org/abs/2606.01414
- Authors: Binxiao Xu, Ruichuan An (Peking University), Bocheng Zou (University of Wisconsin–Madison), Hang Hua (MIT-IBM Watson AI Lab)
The root cause: text-only skills lose spatial evidence and visual boundaries when compressing high-dimensional visual information. Text tells the model *what to do*, but not *where to look* or *how to verify*.
Visual Skill: Making Visual Structure a First-Class Asset
A Visual Skill extends text skills into reusable multimodal skill entities with three components:
The key design principle: visual content is not decoration for a skill — it is the skill's structure itself, encoding not just "what to do" but "where to do it" and "how to verify results".
Three Forms of Visual Skills
Static priors: cross-task spatial references
Stable spatial conventions, like "the submit button is always bottom-right" or "the menu bar is fixed at the top edge", reusable across skills. They resolve static protocol ambiguity by providing a reference map when text instructions are spatially imprecise.Dynamic priors: in-situ state tracking
When elements change position or appearance during operation, dynamic priors update visual memory in real time — e.g. "after clicking to expand, the menu slides out from the top, with new options in the Y=240–400 region". The model *sees* changes rather than *reading* vague descriptions like "the menu expanded".Interleaved visual skills: text steps bound to visual evidence
The most complete form: every text step is bound to its source visual evidence (screenshot, region, interface state at the moment of action). This provides traceability — on failure, the model can trace back to the original evidence to determine whether the button moved, its style changed, or the page never loaded. Text-only skills can only guess blindly.AutoVisualSkill: Automatic Skill Construction
Since manually building Visual Skills is too costly, the paper introduces the AutoVisualSkill pipeline:
1. Diagnose text bottlenecks: identify steps where missing visual information causes ambiguity 2. Generate visual components: extract key visual evidence from trajectory screenshots — element positions, region boundaries, before/after state comparisons 3. Package skill artifacts: combine text logic and visual assets into the standard Visual Skill format with binding protocols
Visual Skills thus become a standard output automatically extractable from any agent trajectory, not a handcrafted expert asset.
Experiments
Whenever tasks require spatial correspondence, visual evidence, or state-aware interaction, visual skills consistently outperform text-only ones.
Implications for the Agent Ecosystem
The paper exposes an overlooked infrastructure problem: current agent skill libraries are entirely text-based, while agent tasks are increasingly multimodal. From web agents to mobile agents to robotic agents, tasks are inherently vision-driven; without visual skills, agents' capability ceilings are locked at the text level.
The idea extends further:
Limitations and Challenges
Solutions will require smarter compression, more robust matching, and more flexible generalization.