MMSkills Deep Dive: When AI Learns to Work While Reading the Manual
> Paper: MMSkills: Towards Multimodal Skills for General Visual Agents > Authors: Kangning Zhang, Shuai Shao, et al. (Shanghai Jiao Tong University + Xiaohongshu) > arXiv: 2605.13527v2 [cs.AI] > Project page: https://deepexperience.github.io/MMSkills/ > GitHub: https://github.com/DeepExperience/MMSkills > HuggingFace dataset: https://huggingface.co/datasets/zhangkangning/mmskills
1. A Seemingly Simple Problem
Ask an AI to complete a desktop task — say, drawing a bar chart in Excel. The traditional approach is a text tutorial: "Step 1, click here; Step 2, select this." But what happens when the actual UI state differs from the tutorial screenshots? A dialog blocks the menu, the data range shifts, a different color theme hides the button?
This is the core pain point of visual agents: **procedural knowledge isn't just *what* to do, but *when — under which visual state* — to do it.
MMSkills, from Shanghai Jiao Tong University and Xiaohongshu, gives the AI not a plain-text manual but a state-sensitive, image-anchored operating handbook that knows when to apply itself.
2. What Does an MMSkill Look Like?
If a traditional text skill is a text-only assembly manual, an MMSkill is an Ikea manual with photos. Each MMSkill package contains:
- Descriptor: e.g., "Create a bar chart from selected data in Excel"
- Textual procedure: numbered steps (select data → Insert → Chart → Bar)
- Runtime state cards**: for each state — when to use, when *not* to use, visual cues, and verification checks (e.g., "cells highlighted blue, status bar shows row count; not applicable when nothing is selected")
- Multi-view keyframes: full-window panorama, focus crops of key regions, and before/after comparisons
- The main agent, seeing the live interface, decides whether a skill might apply and issues a
consultcall - A temporary branch selects relevant state cards and views, aligns them with the *live* interface, and returns structured guidance: applicability judgment, sub-goal, plan, constraints, and verification checks
- The main agent then acts based on the real-time interface, not raw screenshots
- Text skills vs multimodal skills: text suits unambiguous, purely logical operations; MMSkills suit tasks where the visual state decides the next step — a lifeline for small models.
- Branch loading insight: never let reference material pollute the main context; digest it in an isolated space and return structured decision recommendations.
- For vibecoding / computer-use automation: don't write text-only skills when operations depend on UI state — add screenshots, give each state explicit "when to use / when not to use" conditions, and isolate skill references via branch logic.
- Project page: https://deepexperience.github.io/MMSkills/
- GitHub: https://github.com/DeepExperience/MMSkills
- Dataset: https://huggingface.co/datasets/zhangkangning/mmskills
The key design is not "screenshots pasted next to text" but state-driven procedure packages: the agent can itself judge which state the current interface matches and which step applies.
3. Where Do Skills Come From? An Automated Five-Phase Pipeline
Input: many human/AI computer-use and gameplay video trajectories (non-test data). Output: a structured multimodal skill library.
1. Phase 0 — Embedding + clustering: group similar tasks 2. Phase 1 — Skill planning: an LLM proposes atomic skills per cluster 3. Phase 2 — Skill merging: deduplicate, generalize, drop overly broad "umbrella skills" 4. Phase 3 — Text drafting: write the descriptor, procedure, and state-card skeletons *without* looking at images 5. Phase 4 — Image anchoring + audit: select keyframes, annotate focus regions, build multi-view packages, audit quality with a "meta-skill factory"
Key insight: text is written *before* examining images — the textual procedure is fixed first, then visual evidence is anchored from trajectories. This is more efficient and avoids over-anchoring to specific screenshots.
4. Branch Loading: Preventing "Copying the Answer"
Naively injecting retrieved skills into the main context causes two problems:
1. Context explosion — many screenshots eat the token budget 2. Visual anchoring — the agent plans against the reference screenshots instead of the live interface
Branch loading works instead like this:
The skill evidence is "digested" in an isolated branch; the main agent receives a structured decision recommendation, not the raw reference material.
5. Experimental Results
Evaluated on 4 benchmarks (OSWorld desktop, macOSWorld, VAB-Minecraft, Super Mario) with 6 models (Gemini 3.1 Pro/Flash, Qwen3-VL-235B/8B, GLM-5V, Kimi-K2.6).
OSWorld success rates (%)
| Model | No skill | Text-only skill | MMSkills | Gain | |-------|---------|-----------------|----------|------| | Gemini 3.1 Pro | 44.08 | 40.76 | 50.11 | +6.03 | | Gemini 3 Flash | 36.65 | 40.27 | 47.97 | +11.32 | | Qwen3-VL-235B | 21.34 | 28.57 | 39.17 | +17.83 | | GLM-5V | 28.71 | 36.61 | 38.51 | +9.80 | | Kimi-K2.6 | 34.98 | 39.66 | 46.59 | +11.61 | | Qwen3-VL-8B | 10.78 | 14.93 | 25.40 | +14.62 |
Three key findings:
1. Text-only skills can actively hurt — Gemini 3.1 Pro dropped from 44.08% to 40.76%. Procedures without visual state matching can mislead agents into acting at the wrong time. 2. Small models benefit most — Qwen3-VL-8B jumped from 10.78% to 25.40% (+136% relative). External visual procedural knowledge compensates for weak internal priors. 3. Gains are across the board — Chrome, GIMP, VLC, VS Code, Mail, and more.
Ablations
| Variant | Effect | |---------|--------| | Full MMSkills | ✅ best | | Remove state cards | ❌ drop (cannot distinguish relevant states) | | Remove images | ❌ drop (loses visual anchoring) | | Direct loading (no branch) | ❌ harmful (pollutes main context) | | Branch loading without view selection | ⚠️ helps but worse than full version |
Consultation rate and efficiency
| Model | Benchmark | Text skill call rate | MMSkills call rate | Step change | |-------|-----------|----------------------|--------------------|-------------| | Qwen3-VL-235B | OSWorld | 37.50% | 65.28% | 15.22 → 9.87 (−5.35) | | Qwen3-VL-235B | VAB-Minecraft | 54.31% | 64.66% | 34.74 → 27.07 (−7.67) |
Multimodal skills make agents more willing to consult external knowledge (call rate roughly doubled) while shortening trajectories — fewer detours.
6. Practical Takeaways
7. Limitations and Outlook
1. Generation quality depends on public trajectories — buggy trajectories yield buggy skills 2. Cross-domain generalization under-tested — do skills generated on Ubuntu transfer to macOS? 3. Compute overhead — branch loading requires extra inference calls 4. Skill library maintenance — keyframes may go stale after UI updates
The direction is clear: next-generation visual agents won't rely only on internal model knowledge, but will carry an operating manual — not text-only, but image-anchored, state-sensitive, and self-judging.
8. One-Sentence Summary
MMSkills shows that teaching AI to work means telling it not just *what* to do, but *what to see before doing it*, *what means it shouldn't do it*, and *what it should see afterwards*. It sounds like common sense — but most AI skill libraries don't do it, and doing it can double small-model performance across 260+ experiments.
References
Zhang, K., Shao, S., Li, Q., Lin, J., Fu, L., Wang, S., Jiao, W., Lu, Y., Liu, W., Zhang, W., & Yu, Y. (2026). MMSkills: Towards Multimodal Skills for General Visual Agents. *arXiv preprint* arXiv:2605.13527v2.