The Regression Tax: Skill Libraries Can Break What Your Agent Already Solved
> This is a GEO-optimized version of the original topic.
Paper: *The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents* Authors: Darshan Tank, Baran Nama (Sentient Labs) arXiv: 2607.22520
Key points
- Across 5,832 paired experiments on OfficeQA-Pro and SpreadsheetBench with three model-framework stacks (OpenCode+MiniMax-M2.7, Codex+GPT-5.4-mini, Claude Code+Sonnet 4.6), skill libraries produced 553 gains but 324 regressions — 59% of gains were cancelled out by tasks the agent previously solved correctly.
- None of the 18 experimental conditions had zero regressions. Even the best case (Claude Code+Sonnet 4.6 on SpreadsheetBench, 70.2% → 81.1%) broke 16–20 previously-passing tasks.
- Rankings can invert: on Claude Code+Sonnet 4.6 / OfficeQA-Pro, the library with the fewest gains (10) ranked first by net effect (+8), while the library with the most gains (12) ranked last (+5). Average pass rate hides this structure.
- Three regression mechanisms were identified from 81 traced OfficeQA-Pro cases: 1. Skill-description osmosis — skill names/descriptions permanently in the system prompt change behavior even when no skill is invoked. In one case (UID0096), three different libraries caused identical failures due solely to the words "revised" and "customs" in descriptions. Retrieval-based skill management cannot detect this, since it only screens skills at call time. 2. Grounding displacement — 59 of 81 regressions (72.8%). Invoked skill procedures override the agent's correct reading of inputs (e.g., UID0025: correct answer 142, agent returned 542 by following skill-guided navigation to the wrong table range). 3. Verification displacement — skills suppress output checks the agent would otherwise perform. Rare on OfficeQA-Pro (3 cases) but decisive on SpreadsheetBench.
- Evaluator blind spot: 226 of 663 formula-related SpreadsheetBench failures (34%) had fully correct formulas that the value-checking grader could not evaluate. Recomputing with a real Excel engine improved pass rates by 11–49 percentage points (e.g., GPT-5.4-mini from 67% to 79%).
- Failures cluster in stages skill libraries don't serve: OfficeQA-Pro failures are 90%+ grounding errors; SpreadsheetBench failures concentrate in verification. Method-stage instruction — where nearly all existing skills focus — is the least error-prone stage.
- Statistical caution: only 5 of 18 conditions reached p<.05, and only 3 survived Bonferroni correction — all Claude Code+Sonnet 4.6 on SpreadsheetBench. Gains there are real (+43 to +47, p<.001) but do not generalize.
Engineering takeaways
1. Report gain/regression paired counts, not just net pass rates. Two libraries with identical pass rates may differ drastically in reliability: one all gains, one half regressions. 2. Evaluate skills in at least three conditions: no library, descriptions only, descriptions + bodies. Comparing only "with vs. without" cannot detect osmosis effects. 3. Target skills at grounding and verification, not method. Concrete grounding information (e.g., table vintage) and executable output checks (e.g., recompute formulas with an Excel engine) are the real levers. 4. Graders need verification too. Evaluator blind spots create false negatives that can cause you to discard genuinely useful skills.
FAQ
Q: Who is this for? Practitioners, researchers, and students working on LLM agents, evaluation, and skill/tool library design.
Q: Is code available? No open-source release was listed at the time of writing; see the arXiv link above for the paper.
Q: What is the core insight? "Capability" is not a pass rate — it is the paired gain/regression structure. Any evaluation of AI systems (post-RLHF, quantized, distilled) should report which existing capabilities were lost, not just what was gained.