More Skills, Worse Agents? A Log-Decay Scaling Law Shows Where to Stop

Overview

| Item | Detail | |------|--------| | Title | The Scaling Laws of Skills in LLM Agent Systems | | Authors | Charles Chen et al. (multi-institutional collaboration) | | arXiv | 2605.16508 (cs.CL, cs.AI) | | Date | May 2026 | | Link | https://arxiv.org/abs/2605.16508 |

This post (translated and summarized) discusses a paper reporting two coupled scaling laws based on 15 frontier LLMs, 1,141 real skills, and 3+ million routing decisions.

1. The Routing Law: More Skills, More Mistakes

• Single-step routing accuracy decays logarithmically with skill-library size N: accuracy ~ a - b × log(N), with R² > 0.97 across all 15 tested models.
• Each order of magnitude added to the library costs a fixed amount of accuracy.
• Failure follows a three-stage cascade as libraries grow:
1. Local skill competition — semantically similar skills (e.g., "check weather forecast" vs. "check historical weather") compete for the request. 2. Cross-family drift — at hundreds of skills, requests drift to statistically "popular" but semantically unrelated skills due to frequency inertia. 3. "Black-hole" capture — overly broad skills (e.g., "general Q&A") absorb any imprecisely matched request, and their dominance grows non-linearly with library size.

2. The Execution Law: Collapse and Recovery Are Governed Together

• Before routing loss dominates, joint routing approximates multiplication: overall success ≈ product of per-step routing accuracies.
• Strong execution can rescue difficult downstream decisions up to ~4x even when routing erred.
• Crucially, the same log-decay slope b controls both pre-execution collapse and downstream recoverability — one intrinsic property of the library explains both.

3. Quantified Results from Law-Guided Optimization

Improvements came without changing base models — only by restructuring the skill library (splitting black-hole skills, explicit anti-ambiguity for similar skills, grouped routing for high-conflict families):

• Routing accuracy: 71.3% → 91.7%
• Routing hijack (requests swallowed by wrong skills): 22.4% → 4.1%
• ClawBench execution pass rate: 49.3% → 61.6%
• ClawMark execution pass rate: 28.4% → 34.5%

4. Core Insight

Agent performance = f(model capability, skill-library structure) — not an additive relationship. A weaker model with a well-designed library can beat a stronger model with a carelessly built one. Skill exposure policy (which skills are visible in which contexts) is decisive.

5. Open Questions (raised by the author)

1. Generalization: do these laws hold for other skill taxonomies (code generation, physical simulation)? 2. Universality of log decay: as N → ∞ the law predicts accuracy approaching zero — does another decay regime emerge at some threshold? 3. Automation: can a meta-model automatically discover optimal skill-library topology, instead of manual/semi-automatic restructuring?

6. Author's Takeaway

We have scaling laws for LLMs (Kaplan et al.) and test-time compute (Snell et al.); this paper provides the missing scaling laws for skill libraries. Adding skills is not free: every order of magnitude requires active complexity management, or your agent degenerates into a slave of one black-hole skill. As the author puts it: *scale is exactly why you need structure*.

References

1. Chen, C. et al. (2026). The Scaling Laws of Skills in LLM Agent Systems. arXiv:2605.16508. 2. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. 3. Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. NeurIPS 2024. 4. Patil, S.G. et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334.