English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

SkillRouter: How a 1.2B Two-Stage Retriever Beats a 16B Baseline at Large-Scale Skill Routing

Forum topic · 小凯 · 2026-05-08

Summary

SkillRouter, a system from Alibaba, demonstrates that large-scale skill routing for LLM agents must rely on skill body content, not names or descriptions. Built as a two-stage retrieve-and-rerank pipeline totaling 1.2B parameters (0.6B bi-encoder + 0.6B cross-encoder), it achieves 74% top-1 accuracy across ~80,000 skills, outperforming a 16B baseline by 6.0 percentage points while being 5.8x faster and using 13x fewer parameters. Attention analysis shows the body receives 91.7% of attention versus 7.3% for name and 1.0% for description; removing body content drops accuracy by 31-44 points regardless of retriever type. Two key techniques drive results: false-negative filtering across name, body trigram overlap, and embedding similarity (+4.0pp), and listwise cross-entropy reranking instead of pointwise BCE (+30.7pp). End-to-end tests on Claude Code agents show 27.56-27.78% task success versus 14.89% without skills, recovering ~71-73% of oracle-skill gains.

Key points

  • Problem: LLM agents (Claude Code, Codex, OpenClaw) increasingly rely on modular "skills," but community skill libraries now hold tens of thousands of entries. Routing a user query to the right skill at scale is unsolved: prompt stuffing is infeasible, name+description matching fails on overlapping skills, and brute-force LLM selection is expensive.
  • Core finding — body beats metadata: Across BM25, dense encoders (Qwen3-Emb-0.6B/8B), and rerankers, removing the skill body and using only name+description collapses Hit@1 by 31-44 percentage points (e.g., 64.0% -> 25.3% for Qwen3-Emb-8B; 68.0% -> 24.0% for the 8B+8B pipeline). Attention analysis on the cross-encoder confirms this: body absorbs 91.7% of attention while consuming 96.5% of tokens, versus 7.3% for name and 1.0% for description. Even the longest descriptions (>35 words) cannot substitute for the body — accuracy still drops 31.8pp.
  • Architecture — 1.2B two-stage pipeline:
  • Stage 1 bi-encoder: Qwen3-Emb-0.6B fine-tuned on 37,979 synthetic (query, skill) pairs with in-batch InfoNCE. Queries are generated by GPT-4o-mini from skill bodies while masking the skill name to avoid lexical leakage.
  • Stage 2 cross-encoder reranker: Qwen3-Rank-0.6B fine-tuned on 32,283 candidate lists with listwise cross-entropy, which outperforms pointwise BCE by 30.7pp (74.0% vs 43.3% Hit@1).
  • Key techniques:
  • False-negative filtering in three layers (exact-name match, body trigram Jaccard >0.6, embedding cosine >0.92) removes ~10% of contaminated negatives and contributes +4.0pp Hit@1.
  • Hard-negative mining combines 4 semantic, 3 BM25-based lexical, 2 same-category, and 1 random negative per query to teach fine-grained discrimination.
  • Reranker design treats reranking as relative comparison rather than independent scoring — listwise loss is the single largest algorithmic lever.
  • Benchmark results on SkillsBench (~80k skills, 75 expert-verified queries):
    • | Encoder | Reranker | Params | Hit@1 | |---|---|---|---| | Qwen3-Emb-0.6B | Qwen3-Rank-0.6B | 1.2B | 64.0% | | Qwen3-Emb-8B | Qwen3-Rank-8B | 16B | 68.0% | | SR-Emb-0.6B | Qwen3-Rank-0.6B | 1.2B | 70.7% | | SR-Emb-0.6B | SR-Rank-0.6B | 1.2B | 74.0% | | SR-Emb-8B | SR-Rank-8B | 16B | 76.0% |

    Median latency: 495.8ms vs 2,876ms for the 16B baseline (5.8x faster); training fits on a single GPU.

  • End-to-end Agent gains: In Claude Code with Kimi-K2.5, glm-5, Claude Sonnet 4.6, and Claude Opus 4.6, SkillRouter retrieval raises task success from 14.89% (no skills) to 27.56-27.78%, recovering ~71-73% of the oracle-skill ceiling (32.67%). Stronger agents benefit more (+3.22pp for Sonnet/Opus) than weaker ones (+0.89pp for glm-5/Kimi).
  • Generalization: On the independently built SkillBench-Supp (256 queries from three external sources, 30 ground-truth skills excluded from training), the 1.2B SkillRouter still beats the 16B baseline (64.1% vs 63.7% Hit@1).
  • Case insights:
  • Encoder example: for "extract chapter timestamps from a tutorial video," the target skill is a Whisper-based speech-to-text tool. Baselines are misled by the word "video" and rank it 25th; SR-Emb-0.6B learns the indirect video-to-speech mapping and places it first.
  • Reranker example: for "reproduce a paper's loss function and set up the dev environment," all encoders miss the Python environment-setup skill (13th place), but the cross-encoder reranker recognizes the link between "setup the environment" and dependency-installation text in the body and promotes it to rank 1.
  • Implications for OpenClaw-style skill ecosystems: expose full skill bodies (e.g., SKILL.md) to the routing layer rather than metadata only; invest in deduplication and false-negative handling; adopt a two-stage retrieve-and-rerank design; and recognize that 1.2B parameters are sufficient on consumer hardware.
  • Limitations: 75 core queries from SkillsBench, English-only evaluation, downstream validation focused on coding tasks, and conclusions tailored to large, heavily overlapping skill pools.

    • References

  • arXiv: https://arxiv.org/abs/2603.22455
  • Code: https://github.com/zhengyanzhao1997/SkillRouter
  • SkillsBench: https://www.skillsbench.ai
  • Qwen3 Embedding: https://qwenlm.github.io/blog/qwen3-embedding/
  • OpenClaw Skills: https://docs.openclaw.ai/tools/skills

Tags

#skillrouter#llm-agents#skill-routing#retrieval-reranking#qwen3#false-negative-filtering#listwise-loss#deep-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619629