More Skills, Dumber Agents? The Logarithmic Decay Law of Skill Libraries
Everyone building agent systems eventually asks the same question: how big should the toolbox be? Too small, and the agent can do nothing. Too big, and it picks the wrong tool. This paper—"The Scaling Laws of Skills in LLM Agent Systems" by Charles Chen et al. (arXiv:2605.16508)—gives a precise answer after testing 15 frontier LLMs on 1,141 real-world skills across 3+ million routing decisions.
Key points
- Routing law: Single-step routing accuracy decays logarithmically with library size: accuracy ~ a − b × log(N), with R² > 0.97 across all 15 models. Each order of magnitude of added skills costs a fixed amount of accuracy.
- Failure cascade: (1) Local competition between semantically similar skills; (2) cross-family drift toward statistically "popular" but irrelevant skills; (3) capture by "black hole" skills—overly broad tools whose share of calls grows non-linearly as libraries expand.
- Execution law: Before routing loss dominates, joint routing is approximately multiplicative across steps. Strong execution can salvage difficult downstream decisions by about 4×. Crucially, the same slope b that governs routing decay also controls pre-execution collapse and recoverability—two expressions of one underlying quantity.
- Law-guided optimization (same base models, restructured libraries only):
- Routing accuracy: 71.3% → 91.7%
- Routing hijacking: 22.4% → 4.1%
- ClawBench execution pass rate: 49.3% → 61.6%
- ClawMark execution pass rate: 28.4% → 34.5%
- Interventions included splitting black-hole skills, explicit disambiguation of near-duplicate skills, and grouped routing for high-conflict families.
- Core insight: Agent performance = f(model capability, skill library structure). A weak model with a well-designed library can beat a stronger model with a carelessly built one. Exposure policy—which skills are visible in which contexts—is decisive.
Open questions
1. Generalization: Were the 1,141 skills drawn from a specific platform? Do the laws hold for code-generation or physics-simulation skill libraries? 2. Limit behavior: Log decay implies routing accuracy approaching zero as N → ∞. Does another decay regime emerge past some threshold? 3. Automation: Splitting, grouping, and disambiguation are currently manual or semi-automatic. Can a meta-model automatically discover the optimal skill library topology?
Verdict
We have scaling laws for LLMs (Kaplan et al., Chinchilla) and for test-time compute (Snell et al.). This paper adds the missing piece: scaling laws for skill libraries. Adding skills is not free—every order of magnitude demands active complexity management, or your agent degenerates into a servant of its vaguest, black-hole tool.
This is not "scale is all you need"—it is scale is exactly why you need structure.
References
1. Chen, C. et al. (2026). The Scaling Laws of Skills in LLM Agent Systems. https://arxiv.org/abs/2605.16508 2. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361 3. Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. NeurIPS 2024 4. Patil, S.G. et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334