Key Points
- The orchestration trilemma: Inference-time orchestration (e.g., GPT-5/Claude) is strong but never accumulates experience; training-time orchestration can learn but cannot scale to frontier models. Skill-MAS proposes a third path: externalize the orchestration policy as an evolvable natural-language artifact that frozen LLMs execute.
- Meta-Skill design: A structured, version-controllable document with three modules:
- *Module 1 (What)*: intent analysis, subtask decomposition, dependency mapping, success criteria.
- *Module 2 (Who)*: agent role profiling, precise instruction design, input context framing.
- *Module 3 (How)*: topology selection (serial, hierarchical, loop), data-flow and state management, executable code generation.
- Closed-loop evolution over R rounds: 1. *Multi-trajectory rollout*: Sample K=5 independent execution trajectories per task under current Meta-Skill \(S^{(r)}\); compute uncertainty \(u_i\) (std of scores) and difficulty \(d_i\) (negative mean score). 2. *Selective reflection*: Fuse \(u_i\) and \(d_i\) into a priority \(p_i\", use an elbow cutoff to pick the top \(j^*\) tasks, then run intra-task contrastive analysis (compare high- vs. low-scoring trajectories) and cross-task synthesis to build an evidence package \(E\). 3. *Targeted patching*: Update the three Meta-Skill modules where weak, preserving scaffolding, and abstracting to generalizable principles rather than task-specific hacks.
- Experiments: Evaluated on DeepResearchBench, Humanity's Last Exam-Math, BrowseComp-Plus (BCP), and VitaBench across Gemini-3.1-Flash, GPT-5.4-Nano, Qwen3.5-Plus, and DeepSeek-V4-Flash. Optimized frozen models surpass the strongest inference-time reranking baselines:
- Transferability findings:
- *Cross-model, same task*: A Meta-Skill evolved on GPT-5.4-Nano lifted Qwen3.5-Plus from 18.45 to 24.40 on the same task.
- *Cross-task, same LLM*: BCP → VitaBench rose from 0 to 13.10; VitaBench → BCP rose from 0 to 23.21.
- *Cross-model and cross-task* (hardest setting): still delivered positive transfer, indicating the Meta-Skill captures domain-general orchestration wisdom rather than model quirks or task hacks.
- Evolution trajectory on BCP (illustrative): Modules evolved through decomposition tuning → agent-level epistemic controls (e.g., weighted-consensus protocols) → system-level resilience features (backtracking, dynamic replanning, link-validation tasks, merged-node re-execution rights). The path is layered and directional, not random search.
- Externalize orchestration policies into versioned structured docs (SKILL.md style); treat the policy as a first-class artifact rather than burying it in prompts or weights.
- Use multi-trajectory sampling (K≥5); single-trajectory success can be luck, while statistics separate real capability from execution noise.
- Prefer selective reflection over full retrospection: rank tasks by combining uncertainty and difficulty, and concentrate optimization on the most informative ones.
- Abstract findings across tasks into general principles to enable cross-task transfer.
- Selective reflection currently needs ground-truth labels to score trajectories; LLM-as-a-judge self-supervised evaluation is a natural next step.
- Multi-task co-evolution of Meta-Skills is not yet systematically optimized.
- Meta-Skills raise a deeper question: if all orchestration strategies can be externalized into natural language, where does LLM "intelligence" reside — in executing a strategy or in generating one? Skill-MAS's stance is that both matter, but the strategy itself can evolve independently of the executor, decoupling the algorithm from the hardware.
- Lin et al. (2026). *Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems*. Ant Group & HKUST(GZ). arXiv:2606.18837.
| Model | Best Baseline | Skill-MAS Init | Skill-MAS Optimized | Gain | |---|---|---|---|---| | Gemini-3.1-Flash | 21.29 | 21.68 | 29.49 | +38.5% | | GPT-5.4-Nano | 24.83 | 19.64 | 27.55 | +11.0% | | Qwen3.5-Plus | 32.23 | 32.61 | 38.41 | +19.2% | | DeepSeek-V4-Flash | 35.70 | 33.72 | 41.05 | +15.0% |
Cost profile: training-time MAS is cheapest but weakest; inference-time MAS is most expensive per sample; Skill-MAS pays one-time evolution cost then zero marginal overhead, while achieving the highest performance.