> When an AI agent clicks the wrong button on a GUI or miscounts objects in an image, the problem may not be model capability but how skills are stored — pure text simply cannot hold visual information.
The Blind Spot of Text Skills
Today's agent skill learning is essentially note-taking: operation steps written as text instructions, reasoning recorded as text trajectories, successful experiences summarized as text digests. This works for text-heavy tasks but breaks down on visually intensive ones:
- GUI localization: describing a button as "the second icon in the top-right corner" fails across different resolutions and themes
- Dense object counting: for an image with 47 birds, text can only say "count every bird carefully" — it cannot provide spatial grounding to avoid double-counting or omissions
- Visual state tracking: after an operation, text logs "the page refreshed," but cannot encode which elements changed or where
- Code skills: preserve not just snippets but their IDE location, file structure, and debugger state screenshots
- Document skills: preserve paragraph screenshots, table regions, and chart locations alongside summaries
- Multimodal skills: audio waveforms, video keyframes, 3D model viewpoints, and more
- Visual storage cost: screenshots and region annotations take far more space than text
- Matching robustness: theme, resolution, and dynamic content changes affect prior-matching accuracy
- Generalization: visual priors trained on one site may fail on structurally different sites
- Paper: *Agent Skills Should Go Beyond Text: The Case for Visual Skills*
- arXiv: https://arxiv.org/abs/2606.01414
- Authors: Binxiao Xu, Ruichuan An (Peking University), Bocheng Zou (University of Wisconsin), Hang Hua (MIT-IBM Watson AI Lab)
The root cause is straightforward: pure-text skills lose spatial evidence and visual boundaries when compressing high-dimensional visual information. Text tells a model *what* to do, but not *where to look* or *how to verify*.
Visual Skill: Making Visual Structure a First-Class Asset
The paper introduces Visual Skill — extending traditional text-only skills into reusable multimodal skill entities with three components:
1. Declarative text logic: semantic reasoning and flow control — the conventional "step 1, step 2, step 3." 2. Reusable visual priors / source references: the new core. Instead of describing visual information, it directly preserves visual evidence — screenshots, page regions, element bounding boxes, visual state snapshots — loaded on demand as skill "attachments." 3. Multimodal binding protocol: specifies how text steps and visual assets are jointly retrieved and executed. For step 3, "click the submit button," the protocol points to that button's bounding box in a screenshot rather than guessing from a text description.
The key design point: visuals are not decoration for a skill — they are part of the skill's structure. A Visual Skill encodes not only "what to do" but "where to do it" and "how to verify the result."
Three Forms of Visual Skills
Static priors: shared spatial references across tasks
Static priors capture stable spatial conventions — e.g., a web app's submit button is always bottom-right, the menu bar fixed at the top. They resolve static protocol ambiguity by providing a "reference map" when textual spatial descriptions are imprecise.
Dynamic priors: in-situ spatial tracking
Dynamic priors solve state tracking. When interface elements change position or appearance during interaction, visual memory is updated in real time — e.g., "after clicking to expand, the menu slides out from the top, with new options appearing in the Y=240–400 region." Instead of reading that "the menu expanded," the model can *see* before/after visual states.
Interleaved visual skills: text steps bound to visual evidence
The most complete form: every text step is bound to its source visual evidence (screenshots, page regions, interface state at the moment of action). This enables traceability — when execution fails, the model can trace back to the original evidence to determine whether the button moved, its style changed, or the page never loaded. Text-only skills leave the model guessing blindly.
AutoVisualSkill: An Automatic Construction Pipeline
Manually building Visual Skills is too costly, so the paper proposes the AutoVisualSkill pipeline:
1. Diagnose text bottlenecks: analyze which task steps are ambiguous due to missing visual information 2. Generate visual components: extract key visual evidence from trajectory screenshots — element locations, region boundaries, before/after state comparisons 3. Package skill artifacts: combine text logic and visual assets into a standard Visual Skill format with the multimodal binding protocol
Visual Skills thus become a standard output extractable from any agent trajectory, not an expert-only asset.
Experiments: Visual Skills Beat Text-Only Skills
GUI grounding: Visual Skills consistently improve localization accuracy. Text says "click the settings icon in the top right," but icon position varies with resolution; a Visual Skill stores the icon's bounding box from the original screenshot, allowing re-localization by visual similarity.
Dense object counting: by storing visual markers of "already counted regions," Visual Skills avoid double counting and omissions — evidence that text cannot provide.
Whenever tasks require spatial correspondence, visual evidence, or state-aware interaction, visual skills consistently outperform text-only ones.
Implications for the Agent Ecosystem
The paper highlights an overlooked infrastructure problem: current agent skill libraries are entirely text-based, while more and more agent tasks are multimodal — web agents, mobile agents, robotics agents are all fundamentally vision-driven. If skill systems don't support vision, agent capability is capped at the text level.
The idea extends further:
Limitations and Challenges
Directions: smarter compression, more robust matching, more flexible generalization.