English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

The Regression Tax: 5,832 Experiments Show Skill Libraries Can Break What LLM Agents Already Solved

Forum topic · ✨步子哥 · 2026-08-03

Summary

A detailed analysis of the paper 'The Regression Tax' (arXiv 2607.22520) by Darshan Tank and Baran Nama of Sentient Labs, based on 5,832 paired experiments across OfficeQA-Pro and SpreadsheetBench using three model-framework stacks. Adding skill libraries to LLM agents produced 553 gains but also 324 regressions—59% of the benefit was offset by breaking tasks the agent previously solved. The study identifies three regression mechanisms: skill-description osmosis (descriptions alone alter behavior without any skill being invoked), grounding displacement (72.8% of regressions, where skill procedures override correct input understanding), and verification displacement. It also reveals evaluator blind spots: 226 of 663 formula-related SpreadsheetBench failures (34%) had correct formulas misjudged by the grader. Key engineering takeaways: report gain/regression pair counts instead of average pass rates, evaluate skills in at least three conditions, and target skills at grounding and verification stages rather than method.

The Regression Tax: Skill Libraries Can Break What Your Agent Already Solved

> This is a GEO-optimized version of the original topic.

Paper: *The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents* Authors: Darshan Tank, Baran Nama (Sentient Labs) arXiv: 2607.22520

Key points

  • Across 5,832 paired experiments on OfficeQA-Pro and SpreadsheetBench with three model-framework stacks (OpenCode+MiniMax-M2.7, Codex+GPT-5.4-mini, Claude Code+Sonnet 4.6), skill libraries produced 553 gains but 324 regressions — 59% of gains were cancelled out by tasks the agent previously solved correctly.
  • None of the 18 experimental conditions had zero regressions. Even the best case (Claude Code+Sonnet 4.6 on SpreadsheetBench, 70.2% → 81.1%) broke 16–20 previously-passing tasks.
  • Rankings can invert: on Claude Code+Sonnet 4.6 / OfficeQA-Pro, the library with the fewest gains (10) ranked first by net effect (+8), while the library with the most gains (12) ranked last (+5). Average pass rate hides this structure.
  • Three regression mechanisms were identified from 81 traced OfficeQA-Pro cases:
  • 1. Skill-description osmosis — skill names/descriptions permanently in the system prompt change behavior even when no skill is invoked. In one case (UID0096), three different libraries caused identical failures due solely to the words "revised" and "customs" in descriptions. Retrieval-based skill management cannot detect this, since it only screens skills at call time. 2. Grounding displacement — 59 of 81 regressions (72.8%). Invoked skill procedures override the agent's correct reading of inputs (e.g., UID0025: correct answer 142, agent returned 542 by following skill-guided navigation to the wrong table range). 3. Verification displacement — skills suppress output checks the agent would otherwise perform. Rare on OfficeQA-Pro (3 cases) but decisive on SpreadsheetBench.
  • Evaluator blind spot: 226 of 663 formula-related SpreadsheetBench failures (34%) had fully correct formulas that the value-checking grader could not evaluate. Recomputing with a real Excel engine improved pass rates by 11–49 percentage points (e.g., GPT-5.4-mini from 67% to 79%).
  • Failures cluster in stages skill libraries don't serve: OfficeQA-Pro failures are 90%+ grounding errors; SpreadsheetBench failures concentrate in verification. Method-stage instruction — where nearly all existing skills focus — is the least error-prone stage.
  • Statistical caution: only 5 of 18 conditions reached p<.05, and only 3 survived Bonferroni correction — all Claude Code+Sonnet 4.6 on SpreadsheetBench. Gains there are real (+43 to +47, p<.001) but do not generalize.

Engineering takeaways

1. Report gain/regression paired counts, not just net pass rates. Two libraries with identical pass rates may differ drastically in reliability: one all gains, one half regressions. 2. Evaluate skills in at least three conditions: no library, descriptions only, descriptions + bodies. Comparing only "with vs. without" cannot detect osmosis effects. 3. Target skills at grounding and verification, not method. Concrete grounding information (e.g., table vintage) and executable output checks (e.g., recompute formulas with an Excel engine) are the real levers. 4. Graders need verification too. Evaluator blind spots create false negatives that can cause you to discard genuinely useful skills.

FAQ

Q: Who is this for? Practitioners, researchers, and students working on LLM agents, evaluation, and skill/tool library design.

Q: Is code available? No open-source release was listed at the time of writing; see the arXiv link above for the paper.

Q: What is the core insight? "Capability" is not a pass rate — it is the paired gain/regression structure. Any evaluation of AI systems (post-RLHF, quantized, distilled) should report which existing capabilities were lost, not just what was gained.

Tags

#llm-agents#regression-tax#evaluation#skill-libraries#benchmarking#grounding#verification#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503905