Qwen-MM-Plugins: Alibaba Makes Multimodal Agent Capabilities a Pluggable Protocol Layer
On August 10, the Qwen team (QwenLM organization) launched the Qwen-MM-Plugins repository on GitHub under the Apache-2.0 license. It is not a model, not a UI, not a harness—it is a protocol plugin layer that makes any agent harness multimodal-native. The core promise: "Make any agent harness multimodal-native". Same-day commit records show feat(omni-av): add Qwen-Omni audio/video understanding capability, adding Qwen-Omni's audio/video understanding as a new capability plugin.
This means the Qwen team did not miss the August protocol-layer arms race around Agent Plugins 1.0.0 (topicId 178603067)—they differentiated via the multimodal capability plugin track. Vercel's Agent Plugins 1.0.0 standardizes the "protocol layer"; Qwen-MM-Plugins standardizes "multimodal capabilities". Together, they form the full picture of "any agent harness can call any AI capability out of the box".
Architecture: Skill + MCP Server
Each capability plugin = a Skill (so the model knows the tool exists) + an MCP Server (the tool implementation). The key benefit is harness decoupling: without touching the code of Claude Code / Codex / Qoder / OpenClaw / Qwen Code / Gemini CLI / opencode / pi / QwenPaw, adding one MCP config makes these capabilities visible.
The Seven Capability Plugins
- core (basic vision) — the default-installed core plugin. Features: dynamic-resolution image reading (4K screenshots, document pages, 3D models, PDFs auto-split per the VL model's patch grid, no manual resizing), OCR, grounding (bounding boxes), segmentation, ASR, visual chat, web search (requires a Serper API key), image search, and media info metadata reading.
- video-memory (long-video memory) — a hierarchical graph memory mechanism supporting QA over very long videos (>2 hours). Memory is a layered graph rather than a linear KV cache, so query efficiency stays stable as video length grows.
- omni-av (Qwen-Omni audio/video understanding) — added August 10. Seven core tools: omni_asr (plain-text transcription), omni_asr_timestamped (sentence/word-level timestamps with SRT subtitles), omni_multi_speaker_asr (multi-speaker identification and separation), omni_av_caption (time-segmented audio/video description), omni_av_grounding (locate when an event occurs via text query), omni_av_counting (count events/actions/objects), omni_music_caption (genre/mood/instrument/key/vocal tags).
- video-edit (video editing + generation) — video editing workflow plus image/video/audio generation tool integrations.
- blender (3D modeling) — a 22-tool remote driver for Blender. Thin-client mode: local Blender installation required, driven via Python.
- freecad (parametric CAD) — a 14-tool FreeCAD remote driver, also thin-client, with STEP/STL import/export and FEM analysis.
- edu-agent (educational explainer videos) — turns math/science problems into Chinese explainer videos. Pure skill mode, no MCP—the capability is a prompt template given directly to the agent.
- Past 12 months — multimodal capability was a model's "native feature". Qwen-VL, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet each built it in; developers had to rewrite tool-calling code per model.
- Past 6 months — the MCP protocol began standardizing "tool calling", but multimodal capability remained model-bound—choosing Qwen-VL vs. GPT-4V determined what you could read or understand.
- Qwen-MM-Plugins' answer — split Qwen's multimodal capabilities into 7 independent plugins, each exposed via MCP. Harnesses call them with one MCP config, without being locked to the Qwen model.
- "AI coding tools seeing images" gets a standard for the first time — previously Claude Code, Codex, and opencode each had their own image-reading mechanisms; Qwen-MM-Plugins makes it an MCP tool (
vision_chat). - "AI coding tools editing Blender" gets a solution for the first time — 22 thin-wrapped Blender tools via MCP config.
- "AI coding tools watching long videos" gets a memory model for the first time — the video-memory plugin's hierarchical graph memory grows logarithmically.
- Portability of multimodal capability is protocol-guaranteed for the first time — switch from Claude Code to Codex by changing only the MCP config.
- License: Apache-2.0 overall, with MIT-licensed third-party code inside the Blender and FreeCAD plugins (they call Blender/FreeCAD Python APIs directly). Fully free for developers.
- One-line install:
curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bashinstalls the core plugin; then each harness needs one MCP config. - WSL2 is mandatory on Windows — clone the repo in the WSL home directory, not on /mnt/c mounts. No native Windows support out of the box.
- Cookbooks are TBD — only the
corecookbook is complete at launch; other capabilities' demos await content. - 6 system dependencies — ffmpeg, libreoffice, blender, texlive, chromium. A full Mac setup needs ~2–3GB disk and about half an hour.
- Depends on Qwen model APIs — DASHSCOPE_API_KEY is required; Serper is optional. Qwen-MM-Plugins is thus a capability wrapper around the Qwen model ecosystem, not fully independent.
- 8-08 Agent Plugins 1.0.0 (topicId 178603067) — Vercel-led protocol-layer standardization.
- 8-09 Microsoft SkillOpt (topicId 178603080) — portable experience layer.
- 8-10 OpenChamber (Topic 1) — Harness / Runtime / UI three-layer decoupling.
- 8-10 Qwen-MM-Plugins (this post) — protocolized multimodal capability layer.
- Qwen-MM-Plugins GitHub: https://github.com/QwenLM/Qwen-MM-Plugins
- Chinese README: https://github.com/QwenLM/Qwen-MM-Plugins/blob/main/README.zh.md
- Announcement: https://x.com/Alibaba_Qwen/status/2086664887560970531
- Installation docs: https://github.com/QwenLM/Qwen-MM-Plugins/blob/main/docs/en/installation.md
- Cookbooks: https://github.com/QwenLM/Qwen-MM-Plugins/tree/main/cookbooks
- Install script: https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh
- DashScope API key: https://dashscope.console.aliyun.com/apiKey
- Serper API key: https://serper.dev/
Why It Matters: From Model-Built-In to Protocol-External
Concrete wins for AI coding workflows:
Notable Details
Context: August's Full-Stack Decoupling
August 10 marks a key moment in AI coding toolchain full-stack decoupling—harness (OpenCode), runtime (OpenChamber), capability layer (Qwen-MM-Plugins), experience layer (SkillOpt), protocol layer (Agent Plugins 1.0.0), model layer (Muse Glimmer). Each layer now has an independently replaceable open-source implementation: "AI coding tools" shift from a single vendor's product to an assembly of swappable parts.