On August 10, the Qwen team (QwenLM organization) launched the Qwen-MM-Plugins repository on GitHub under the Apache-2.0 license. It is not a model, a UI, or a harness — it is a protocol plugin layer whose core promise is "Make any agent harness multimodal-native." A same-day commit (feat(omni-av): add Qwen-Omni audio/video understanding capability) added Qwen-Omni's audio/video understanding as a new capability plugin.
This positions Qwen firmly in the August protocol-layer arms race for AI coding toolchains, following the August 8 Agent Plugins 1.0.0 release led by Vercel. Where Agent Plugins standardizes the protocol layer, Qwen-MM-Plugins standardizes the multimodal capability layer — together they form the full picture of "any agent harness calling any AI capability out of the box."
Architecture: Skill + MCP Server
Each capability plugin combines a Skill (so the model knows the tool exists) with an MCP Server (the tool's implementation). This decouples the harness: no changes to Claude Code / Codex / Qoder / OpenClaw / Qwen Code / Gemini CLI / opencode / pi / QwenPaw are needed — just one MCP configuration.
The seven capability plugins
- core (base vision) — installed by default. Dynamic-resolution image reading (4K screenshots, document pages, 3D models, PDFs auto-tiled to the VL model's patch grid, no manual resize), OCR, grounding (bounding boxes), segmentation, ASR, vision chat, web search (requires a Serper API key), image search, and media metadata.
- video-memory (long-video memory) — hierarchical graph memory supporting QA over very long videos (>2 hours). Because memory is a layered graph rather than a linear KV cache, query efficiency stays stable as video length grows.
- omni-av (Qwen-Omni AV understanding) — added August 10. Seven tools:
omni_asr,omni_asr_timestamped(SRT, sentence/word-level),omni_multi_speaker_asr,omni_av_caption,omni_av_grounding,omni_av_counting, andomni_music_caption(genre/mood/instrument/key/vocal tags). - video-edit — video editing workflows plus image/video/audio generation tool integration.
- blender (3D modeling) — a 22-tool remote driver for Blender in thin-client mode (local Blender required, driven via Python).
- freecad (parametric CAD) — a 14-tool FreeCAD driver, also thin-client, with STEP/STL import/export and FEM analysis.
- edu-agent (educational explainer videos) — turns math/science problems into Chinese explainer videos. Skill-only, no MCP — the capability is delivered as prompt templates directly to the agent.
- Reading images via coding tools gets a standard — previously Claude Code, Codex, and opencode each had their own image-reading mechanisms. Now
vision_chatis one MCP tool any harness can consume. - Editing Blender scenes gets a solution — a 22-tool thin wrapper replaces hand-written Python servers.
- Watching long videos gets a memory model — hierarchical graph memory instead of full-frame context stuffing or lossy summarization.
- Multimodal portability is protocol-guaranteed — switch from Claude Code to Codex by swapping MCP configuration only, no code changes.
- Licensing: Apache-2.0 overall, with MIT-licensed third-party code in the Blender and FreeCAD plugins (they call Blender/FreeCAD Python APIs directly).
- One-line install for the core plugin:
curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash, then add an MCP config per harness. - WSL2 is mandatory on Windows (Ubuntu recommended; clone into the WSL home, not under
/mnt/c). No native Windows support. - Cookbooks are mostly TBD — only the
corecookbook is complete at launch. - Six system dependencies —
ffmpeg,libreoffice,blender,texlive,chromium; a full Mac setup needs ~2–3 GB and about half an hour. - Qwen model dependency — a
DASHSCOPE_API_KEYis required, so this is a Qwen-ecosystem capability wrapper rather than a fully model-independent layer. - Aug 8 — Agent Plugins 1.0.0 (Vercel-led protocol layer standardization)
- Aug 9 — Microsoft SkillOpt (portable experience layer)
- Aug 10 — OpenChamber (Harness / Runtime / UI three-layer decoupling)
- Aug 10 — Qwen-MM-Plugins (multimodal capability layer protocolization)
- Qwen-MM-Plugins GitHub: https://github.com/QwenLM/Qwen-MM-Plugins
- Chinese README: https://github.com/QwenLM/Qwen-MM-Plugins/blob/main/README.zh.md
- Announcement: https://x.com/Alibaba_Qwen/status/2086664887560970531
- Installation docs: https://github.com/QwenLM/Qwen-MM-Plugins/blob/main/docs/en/installation.md
- Cookbooks: https://github.com/QwenLM/Qwen-MM-Plugins/tree/main/cookbooks
- One-line installer: https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh
- DashScope API key: https://dashscope.console.aliyun.com/apiKey
- Serper API key: https://serper.dev/
Why it matters for AI coding workflows
Notable details and caveats
Context: the August toolchain decoupling timeline
Together with independent open-source implementations at the harness, runtime, experience, protocol, and model layers, AI coding tools are shifting from single-vendor products toward assemblable machines made of replaceable parts.