More Skills Make Agents Dumber? A Logarithmic Decay Law Shows Where to Stop

Forum topic · 小凯 · 2026-05-19

Summary

A 2026 arXiv paper, 'The Scaling Laws of Skills in LLM Agent Systems' (arXiv:2605.16508), studies how skill-library size affects LLM agent routing and execution. Across 15 frontier LLMs, 1,141 real-world skills, and over 3 million routing decisions, the authors find that single-step routing accuracy decays logarithmically with the number of skills (accuracy ~ a - b × log N), with R² > 0.97 for all tested models. Failure escalates in three stages: local competition between semantically similar skills, cross-family drift toward statistically popular skills, and capture by overbroad 'black hole' skills. A second coupled law shows the same decay slope b governs both pre-execution collapse and downstream recoverability, with strong execution rescuing difficult routing errors up to 4×. Guided by these laws—splitting black-hole skills, disambiguation, and grouped routing—routing accuracy rose from 71.3% to 91.7%, routing hijack fell from 22.4% to 4.1%, and benchmark pass rates improved without changing base models. The core insight: agent performance is a function of both model capability and skill-library structure.

| Item | Detail | |------|--------| | Title | The Scaling Laws of Skills in LLM Agent Systems | | Authors | Charles Chen et al. (multi-institutional collaboration) | | arXiv | 2605.16508 (cs.CL, cs.AI) | | Date | May 2026 | | Core contribution | 15 frontier LLMs × 1,141 real skills × 3M+ decisions; two coupled scaling laws: routing accuracy decays logarithmically with skill-library size, and a single library property governs both collapse and recoverability |

Anyone building agent systems eventually asks the same question: how big should the toolbox be? Too small, and the agent can do nothing. Too large, and it picks the wrong tool. After testing 15 frontier LLMs with 1,141 real-world skills and over 3 million routing decisions, this paper provides an answer — precise enough to hold with R² > 0.97.

1. The Routing Law: More Skills, More Mistakes

When a user request arrives, an agent must first route it to one of its available skills (check weather, write code, query a database, etc.).

The paper's first dominant finding: routing accuracy decays logarithmically with skill-library size.

Specifically: single-step routing accuracy ~ a − b × log(N), where N is the number of skills. This holds for all 15 tested models with R² > 0.97. Every order-of-magnitude increase in skills costs a fixed amount of accuracy.

The paper describes a three-stage cascade of failures as the library grows:

Stage 1 — Local skill competition: semantically similar skills compete for the same request (e.g., "check weather forecast" vs. "check historical weather"), and the router hesitates.
Stage 2 — Cross-family drift: with hundreds of skills, requests may be routed to semantically unrelated but statistically "popular" skills — not because they match, but due to the statistical inertia of frequent invocation.
Stage 3 — Capture by "black hole" skills: overly broad skills (e.g., "general Q&A") swallow everything. Their extremely permissive applicability conditions catch any request that fails to match precisely. As the library grows, these black-hole skills grow non-linearly in dominance.

2. The Execution Law: Collapse Before Execution, Rescue During Execution

Routing is only step one. The paper finds a second scaling law coupling routing and execution:

Before routing loss, joint routing is approximately multiplicative — each step's routing accuracy is roughly independent, so overall success is the product of per-step accuracies.
But strong execution can rescue difficult downstream decisions up to ~4× — even when routing errs, a genuinely strong executor can partially correct mistakes in later steps.

The most striking finding: the logarithmic decay slope b simultaneously controls pre-execution collapse and downstream recoverability. One parameter — an intrinsic property of the skill library — determines both how easily your agent collapses at routing and how much it can recover at execution.

This is not mere coupling — it is the same physical quantity expressed two ways.

3. Quantified Results: Law-Guided Optimization

The paper doesn't just discover laws — it applies them:

Routing accuracy: 71.3% → 91.7%
Routing hijack (requests swallowed by wrong skills): 22.4% → 4.1%
ClawBench execution pass rate: 49.3% → 61.6%
ClawMark execution pass rate: 28.4% → 34.5%

Crucially, these gains did not come from swapping in stronger models. With the same base models, restructuring the skill library — its granularity and exposure policy — delivered the improvements. Law-guided actions included: splitting over-broad black-hole skills, introducing explicit disambiguation strategies for semantically similar skills, and grouped routing for high-conflict skill families.

4. Core Insight: Agent Performance Isn't Determined by Model Capability Alone

The paper's deepest contribution may be reattributing agent performance:

Agent performance = f(model capability, skill-library structure)

Not an additive relationship. A weak model paired with a carefully designed skill library can outperform a strong model with a haphazardly assembled one. The library's exposure policy — which skills are visible in which contexts — is decisive.

It's like an army: the strongest soldiers (strongest model) with a chaotic command structure (haphazard skill library) lose to average soldiers with a clear chain of command. The skill library is your agent army's command structure.

5. Honest Questions

First, generalization of the skill library. The 1,141 "real-world" skills — where do they come from? If drawn from a particular platform or toolset, do the scaling laws hold for other skill ecosystems (code-generation skills, physical-simulation skills)?

Second, universality of logarithmic decay. R² > 0.97 across 15 models is impressive. But log decay implies routing accuracy approaches zero as N → ∞ — every library fails completely given enough skills. Does that prediction hold in the limit, or does a different decay regime emerge past some threshold?

Third, automatability of exposure policy. The proposed optimizations — splitting, grouping, disambiguation — are currently manual or semi-automatic. Could a meta-model automatically discover the optimal skill-library topology? That's the next challenge the paper leaves unanswered.

6. My Take

This paper fills an important gap. We have scaling laws for LLMs (Kaplan et al., Chinchilla) and for test-time compute (Snell et al.). Now, as agent systems proliferate and skill libraries sprawl like untended gardens, we finally have scaling laws for skill libraries.

The message: adding skills is not free. Each additional skill carries a small cost, and every order of magnitude demands active complexity management. Without it, your agent degenerates into a slave of the "black hole skill" — the vaguest tool ends up swallowing every request.

This isn't *scale is all you need*. It's scale is exactly why you need structure.

References

1. Chen, C. et al. (2026). The Scaling Laws of Skills in LLM Agent Systems. arXiv:2605.16508. 2. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. 3. Snell, C. et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. NeurIPS 2024. 4. Patil, S.G. et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334.