MMSkills Deep Dive: Teaching AI to Work with an Illustrated Manual

> Paper: MMSkills: Towards Multimodal Skills for General Visual Agents > Authors: Kangning Zhang, Shuai Shao et al. (Shanghai Jiao Tong University + Xiaohongshu) > arXiv: 2605.13527v2 [cs.AI] > Project page: https://deepexperience.github.io/MMSkills/ > GitHub: https://github.com/DeepExperience/MMSkills > HuggingFace dataset: https://huggingface.co/datasets/zhangkangning/mmskills

---

1. A Seemingly Simple Problem

Ask an AI to complete a task on a computer—say, drawing a bar chart in Excel. The traditional approach is a text tutorial: "Step 1, click here; Step 2, select that." But what happens when the live interface doesn't match the tutorial's screenshots? A dialog box blocks the menu, the data range moved, a different theme hides the button?

This is the core pain point of visual agents: procedural knowledge isn't just "what to do"—it's "what to do given a specific visual state."

MMSkills, proposed by Shanghai Jiao Tong University and Xiaohongshu, gives agents not a text-only manual, but an illustrated, state-sensitive handbook that knows when it applies and when it doesn't.

---

2. What Does an MMSkill Look Like?

If a text skill is like a word-only assembly guide, an MMSkill is like an Ikea manual with photos:

• Descriptor: e.g., "Create a bar chart from selected data in Excel"
• Textual procedure: numbered steps (select the data range, Insert → Chart → Bar chart, adjust title and styling)
• Runtime state cards: for each state (e.g., "data selected," "chart inserted"), the card specifies *when to use it*, *when NOT to use it*, *visual cues to look for* (highlighted cells, status bar, selection handles), and *how to verify* the state
• Multi-view keyframes: full-window panoramas, focused crops of relevant regions, and before/after comparisons

Key design: it's not "screenshots pasted next to text"—it's a state-driven procedure package. Each state card lets the agent judge which state the current interface matches and which step to take.

---

3. Where Do Skills Come From? An Automatic Five-Stage Pipeline

Writing illustrated manuals by hand would be costly, but MMSkills generates them automatically from public trajectories (human or AI computer/game operation videos, non-test data):

1. Phase 0 – Embedding + clustering: group similar tasks together 2. Phase 1 – Skill planning: an LLM proposes atomic skills per cluster 3. Phase 2 – Skill merging: deduplicate, generalize, remove overly broad "umbrella skills" 4. Phase 3 – Text drafting: write the descriptor, procedure, and state-card framework without looking at images first 5. Phase 4 – Image anchoring + auditing: select keyframes from trajectories, mark focus regions, build multi-view packages, and audit quality via a "meta-skill factory"

Key insight: drafting the text *before* seeing the images keeps the procedure stable and avoids over-anchoring to specific screenshots.

---

4. Branch Loading: Preventing "Copying the Answer"

With illustrated manuals, the next question is how to make the agent *reference* rather than *copy*. Stuffing multi-image skills directly into the main context causes two problems:

1. Context explosion — many screenshots eat the token budget 2. Visual anchoring — the agent plans against the reference screenshots instead of the live interface

Branch loading solves this: when the agent wonders whether a skill applies, it consults a temporary skill branch that selects relevant state cards and views, aligns them with the live interface, and returns structured guidance—applicability judgment, sub-goals, plan, constraints, don'ts, and verification checks. The main agent receives a decision recommendation, not raw screenshots, and still acts based on the real-time interface.

---

5. Experimental Results

Tested on 4 benchmarks (OSWorld desktop, macOSWorld, VAB-Minecraft, Super Mario) with 6 models (Gemini 3.1 Pro/Flash, Qwen3-VL-235B/8B, GLM-5V, Kimi-K2.6).

OSWorld overall success rate (%)

| Model | No skill | Text-only skill | MMSkills | Gain | |---|---|---|---|---| | Gemini 3.1 Pro | 44.08 | 40.76 | 50.11 | +6.03 | | Gemini 3 Flash | 36.65 | 40.27 | 47.97 | +11.32 | | Qwen3-VL-235B | 21.34 | 28.57 | 39.17 | +17.83 | | GLM-5V | 28.71 | 36.61 | 38.51 | +9.80 | | Kimi-K2.6 | 34.98 | 39.66 | 46.59 | +11.61 | | Qwen3-VL-8B | 10.78 | 14.93 | 25.40 | +14.62 |

Three key findings

1. Text-only skills can be harmful — Gemini 3.1 Pro drops from 44.08% to 40.76%. Procedures lacking visual state matching can "misdirect" agents into acting in the wrong state. 2. Small models benefit most — Qwen3-VL-8B jumps from 10.78% to 25.40% (+136% relative). External visual procedural knowledge compensates for weak internal priors. 3. Gains are broad, not domain-specific — improvements across Chrome, GIMP, VLC, VS Code, Mail, and more.

Ablations

• Removing state cards → performance drops (can't distinguish relevant states)
• Removing images → drops (loses visual anchoring)
• Direct loading (no branching) → actively harmful (reference screenshots conflict with the live interface)
• Branch loading without view selection → helps, but less than the full version

Invocation rate and efficiency

| Model | Benchmark | Text-skill invocations | MMSkills invocations | Step change | |---|---|---|---|---| | Qwen3-VL-235B | OSWorld | 37.50% | 65.28% | 15.22 → 9.87 (-5.35) | | Qwen3-VL-235B | VAB-Minecraft | 54.31% | 64.66% | 34.74 → 27.07 (-7.67) |

Multimodal skills make agents far more willing to consult external knowledge (roughly double the invocation rate) while shortening trajectories—fewer detours.

---

6. Practical Takeaways

• Text vs. multimodal skills: text suits unambiguous logical operations; multimodal suits tasks where visual state determines the next step. For small models, multimodal skills are a lifeline.
• Branch-loading heuristic for any "give the agent reference material" scenario: don't pollute the main context; digest references in an isolated space; return structured decision advice, not raw content.
• For vibecoding / automation with visual models: don't write text-only skills when operations depend on UI state—add screenshots; give every state "when to use / when not to use" guidance; isolate skill references with branch logic.

7. Limitations and Outlook

The paper acknowledges:

1. Skill quality depends on the public trajectories used (bugs propagate) 2. Cross-domain generalization is under-validated (Ubuntu-generated skills on macOS?) 3. Branch loading adds inference overhead 4. Skill libraries need maintenance—keyframes go stale as UIs update

Still, the direction is clear: next-generation visual agents will carry operation manuals—illustrated, state-sensitive, self-judging—not just rely on internal model knowledge.

---

8. One-Sentence Summary

MMSkills shows that teaching AI to work means telling it not just "what to do," but "when to do it given what it sees," "when NOT to do it," and "what it should see afterward." Most AI skill libraries miss this; the SJTU + Xiaohongshu team demonstrated—across 260+ experiment groups—that getting it right can double small-model performance.

---

References

Zhang, K., Shao, S., Li, Q., Lin, J., Fu, L., Wang, S., Jiao, W., Lu, Y., Liu, W., Zhang, W., & Yu, Y. (2026). MMSkills: Towards Multimodal Skills for General Visual Agents. *arXiv preprint* arXiv:2605.13527v2.

Project page: https://deepexperience.github.io/MMSkills/ GitHub: https://github.com/DeepExperience/MMSkills Dataset: https://huggingface.co/datasets/zhangkangning/mmskills