More Skills, Dumber Agents? A Logarithmic Decay Law Tells You Where to Stop Scaling Your Skill Library

Forum topic · 小凯 · 2026-05-19

Summary

A paper titled 'The Scaling Laws of Skills in LLM Agent Systems' (arXiv:2605.16508) reports findings from 15 frontier LLMs, 1,141 real-world skills, and over 3 million routing decisions. Its central result: single-step routing accuracy decays logarithmically with skill library size, following accuracy ~ a - b * log(N) with R-squared above 0.97 across all tested models. Failures progress through three stages: local competition between semantically similar skills, drift toward statistically popular but irrelevant skills, and capture by overly broad 'black hole' skills. The same library property governing pre-execution collapse also controls downstream recoverability—strong execution can rescue about 4x of difficult downstream decisions. Guided by these laws, skill library restructuring (splitting black-hole skills, disambiguation, grouped routing) lifted routing accuracy from 71.3% to 91.7%, reduced routing hijacking from 22.4% to 4.1%, and improved execution pass rates on ClawBench and ClawMark—without changing base models. The paper argues agent performance is a joint function of model capability and skill library structure, not model capability alone.

More Skills, Dumber Agents? The Logarithmic Decay Law of Skill Libraries

Everyone building agent systems eventually asks the same question: how big should the toolbox be? Too small, and the agent can do nothing. Too big, and it picks the wrong tool. This paper—"The Scaling Laws of Skills in LLM Agent Systems" by Charles Chen et al. (arXiv:2605.16508)—gives a precise answer after testing 15 frontier LLMs on 1,141 real-world skills across 3+ million routing decisions.

Key points

Routing law: Single-step routing accuracy decays logarithmically with library size: accuracy ~ a − b × log(N), with R² > 0.97 across all 15 models. Each order of magnitude of added skills costs a fixed amount of accuracy.
Failure cascade: (1) Local competition between semantically similar skills; (2) cross-family drift toward statistically "popular" but irrelevant skills; (3) capture by "black hole" skills—overly broad tools whose share of calls grows non-linearly as libraries expand.
Execution law: Before routing loss dominates, joint routing is approximately multiplicative across steps. Strong execution can salvage difficult downstream decisions by about 4×. Crucially, the same slope b that governs routing decay also controls pre-execution collapse and recoverability—two expressions of one underlying quantity.
Law-guided optimization (same base models, restructured libraries only):
Routing accuracy: 71.3% → 91.7%
Routing hijacking: 22.4% → 4.1%
ClawBench execution pass rate: 49.3% → 61.6%
ClawMark execution pass rate: 28.4% → 34.5%
Interventions included splitting black-hole skills, explicit disambiguation of near-duplicate skills, and grouped routing for high-conflict families.
Core insight: Agent performance = f(model capability, skill library structure). A weak model with a well-designed library can beat a stronger model with a carelessly built one. Exposure policy—which skills are visible in which contexts—is decisive.

Open questions

1. Generalization: Were the 1,141 skills drawn from a specific platform? Do the laws hold for code-generation or physics-simulation skill libraries? 2. Limit behavior: Log decay implies routing accuracy approaching zero as N → ∞. Does another decay regime emerge past some threshold? 3. Automation: Splitting, grouping, and disambiguation are currently manual or semi-automatic. Can a meta-model automatically discover the optimal skill library topology?

Verdict

We have scaling laws for LLMs (Kaplan et al., Chinchilla) and for test-time compute (Snell et al.). This paper adds the missing piece: scaling laws for skill libraries. Adding skills is not free—every order of magnitude demands active complexity management, or your agent degenerates into a servant of its vaguest, black-hole tool.

This is not "scale is all you need"—it is scale is exactly why you need structure.

References

1. Chen, C. et al. (2026). The Scaling Laws of Skills in LLM Agent Systems. https://arxiv.org/abs/2605.16508 2. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361 3. Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. NeurIPS 2024 4. Patil, S.G. et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334