| Item | Detail | |------|--------| | Title | The Scaling Laws of Skills in LLM Agent Systems | | Authors | Charles Chen et al. (multi-institutional collaboration) | | arXiv | 2605.16508 (cs.CL, cs.AI) | | Date | May 2026 | | Core contribution | 15 frontier LLMs × 1,141 real skills × 3M+ decisions; two coupled scaling laws: routing accuracy decays logarithmically with skill-library size, and a single library property governs both collapse and recoverability |
Anyone building agent systems eventually asks the same question: how big should the toolbox be? Too small, and the agent can do nothing. Too large, and it picks the wrong tool. After testing 15 frontier LLMs with 1,141 real-world skills and over 3 million routing decisions, this paper provides an answer — precise enough to hold with R² > 0.97.
1. The Routing Law: More Skills, More Mistakes
When a user request arrives, an agent must first route it to one of its available skills (check weather, write code, query a database, etc.).
The paper's first dominant finding: routing accuracy decays logarithmically with skill-library size.
Specifically: single-step routing accuracy ~ a − b × log(N), where N is the number of skills. This holds for all 15 tested models with R² > 0.97. Every order-of-magnitude increase in skills costs a fixed amount of accuracy.
The paper describes a three-stage cascade of failures as the library grows:
- Stage 1 — Local skill competition: semantically similar skills compete for the same request (e.g., "check weather forecast" vs. "check historical weather"), and the router hesitates.
- Stage 2 — Cross-family drift: with hundreds of skills, requests may be routed to semantically unrelated but statistically "popular" skills — not because they match, but due to the statistical inertia of frequent invocation.
- Stage 3 — Capture by "black hole" skills: overly broad skills (e.g., "general Q&A") swallow everything. Their extremely permissive applicability conditions catch any request that fails to match precisely. As the library grows, these black-hole skills grow non-linearly in dominance.
- Before routing loss, joint routing is approximately multiplicative — each step's routing accuracy is roughly independent, so overall success is the product of per-step accuracies.
- But strong execution can rescue difficult downstream decisions up to ~4× — even when routing errs, a genuinely strong executor can partially correct mistakes in later steps.
- Routing accuracy: 71.3% → 91.7%
- Routing hijack (requests swallowed by wrong skills): 22.4% → 4.1%
- ClawBench execution pass rate: 49.3% → 61.6%
- ClawMark execution pass rate: 28.4% → 34.5%
2. The Execution Law: Collapse Before Execution, Rescue During Execution
Routing is only step one. The paper finds a second scaling law coupling routing and execution:
The most striking finding: the logarithmic decay slope b simultaneously controls pre-execution collapse and downstream recoverability. One parameter — an intrinsic property of the skill library — determines both how easily your agent collapses at routing and how much it can recover at execution.
This is not mere coupling — it is the same physical quantity expressed two ways.
3. Quantified Results: Law-Guided Optimization
The paper doesn't just discover laws — it applies them:
4. Core Insight: Agent Performance Isn't Determined by Model Capability Alone
The paper's deepest contribution may be reattributing agent performance:
Agent performance = f(model capability, skill-library structure)
Not an additive relationship. A weak model paired with a carefully designed skill library can outperform a strong model with a haphazardly assembled one. The library's exposure policy — which skills are visible in which contexts — is decisive.
It's like an army: the strongest soldiers (strongest model) with a chaotic command structure (haphazard skill library) lose to average soldiers with a clear chain of command. The skill library is your agent army's command structure.
5. Honest Questions
First, generalization of the skill library. The 1,141 "real-world" skills — where do they come from? If drawn from a particular platform or toolset, do the scaling laws hold for other skill ecosystems (code-generation skills, physical-simulation skills)?
Second, universality of logarithmic decay. R² > 0.97 across 15 models is impressive. But log decay implies routing accuracy approaches zero as N → ∞ — every library fails completely given enough skills. Does that prediction hold in the limit, or does a different decay regime emerge past some threshold?
Third, automatability of exposure policy. The proposed optimizations — splitting, grouping, disambiguation — are currently manual or semi-automatic. Could a meta-model automatically discover the optimal skill-library topology? That's the next challenge the paper leaves unanswered.
6. My Take
This paper fills an important gap. We have scaling laws for LLMs (Kaplan et al., Chinchilla) and for test-time compute (Snell et al.). Now, as agent systems proliferate and skill libraries sprawl like untended gardens, we finally have scaling laws for skill libraries.
The message: adding skills is not free. Each additional skill carries a small cost, and every order of magnitude demands active complexity management. Without it, your agent degenerates into a slave of the "black hole skill" — the vaguest tool ends up swallowing every request.
This isn't *scale is all you need*. It's scale is exactly why you need structure.
References
1. Chen, C. et al. (2026). The Scaling Laws of Skills in LLM Agent Systems. arXiv:2605.16508. 2. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. 3. Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. NeurIPS 2024. 4. Patil, S.G. et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334.