The Regression Tax: Why Skill Libraries Break Tasks LLM Agents Could Already Solve (5832-Experiment Analysis)
Forum topic · ✨步子哥 · 2026-08-03
Summary
Sentient Labs researchers Darshan Tank and Baran Nama ran 5,832 paired experiments across two office automation benchmarks (OfficeQA-Pro, SpreadsheetBench) using three model-framework stacks to measure the true cost of adding skill libraries to LLM agents. Comparing identical tasks with and without skills, they found 553 gains and 324 regressions — meaning 59% of improvement was offset by tasks the agent previously solved correctly. No condition had zero regressions. Three failure mechanisms explain the regressions: skill-description osmosis (descriptions alter behavior even when never invoked), grounding displacement (skill procedures override correct input interpretation, causing 72.8% of OfficeQA-Pro regressions), and verification displacement (skill scripts suppress the agent's output checks). Failures concentrate in grounding (90%+) and verification stages, while existing skills over-index on method. The study also exposes grader blind spots: 226 correctly evaluated formulas were marked wrong. The paper argues agents should be evaluated by gain/regression pair counts, not aggregate pass rates.
Key points
- Central finding — the Regression Tax: Across 5,832 paired trials on OfficeQA-Pro and SpreadsheetBench using OpenCode+M2.7, Codex+GPT-5.4-mini, and Claude Code+Sonnet 4.6, skill libraries produced 553 gains and 324 regressions, offsetting roughly 59% of the headline improvement.
- No condition was regression-free: Even the strongest configuration (Claude Code+Sonnet 4.6 on SpreadsheetBench, 70.2% → 81.1%) broke 16–20 previously correct tasks.
- Rankings invert under net scoring: In one Claude Code+Sonnet 4.6 / OfficeQA-Pro comparison, the library with the lowest gain (10) became the best after subtracting regressions (+8), overtaking the highest-gain library (+5). Aggregate pass rate hides this structure.
- Three regression mechanisms (from 81 hand-traced OfficeQA-Pro failures):
- Skill-description osmosis — descriptions sit in the system prompt and bias behavior even when never invoked. UID0096: the words "revised" and "customs" in three different skill libraries shifted a tariff moving-average answer from 37.708% (correct) to 38.757% in all three, with zero skills actually called.
- Grounding displacement — a skill's procedure overrides the agent's correct interpretation of the input. UID0025: Sonnet 4.6 returned 142 unaided, but after an anthropic skill dispatched a navigation-then-arithmetic path, returned 542. Responsible for 59/81 (72.8%) of OfficeQA-Pro regressions.
- Verification displacement — the skill script suppresses the agent's output check. Rare on OfficeQA-Pro (3 cases) but dominant on SpreadsheetBench, where 226/663 formula failures (34%) were actually correct formulas the grader could not evaluate (structured references); rerunning those through a real Excel engine boosted each library's pass rate by 11–49 points.
- Failures cluster at the pipeline's edges, not the middle: OfficeQA-Pro fails ≥90% at grounding (misreading tables, years, definitions); SpreadsheetBench fails concentrate at verification. The method stage — where existing skills are aimed — is the least problematic.
- Statistical caution: Only 5/18 conditions reach p<.05; after Bonferroni correction, only 3 survive — all Claude Code+Sonnet 4.6 on SpreadsheetBench. Gains elsewhere are indistinguishable from noise.
- Engineering takeaways:
1. Report gain/regression paired counts, not just net pass rate.
2. Evaluate skills under three regimes: no library, description-only, description+body — to separate osmosis from invocation effects. Retrieval-time guardrails (ASSAY, GRASP, RSEA) cannot catch osmosis because descriptions are always live in context.
3. Target grounding and verification skills, not more method recipes. Concrete input-location tools ("what is this table's vintage?") and executable output checks ("recompute this formula in Excel") are the leverage points.
4. Audit graders — SpreadsheetBench's value checker produced 226 false negatives, which a regression-style skill optimizer could silently discard.
- Broader principle: Aggregate pass rate is itself the largest evaluation blind spot for agents. "Capability" should be treated as a gain/regression distribution, not a scalar — the same logic applies to RLHF, quantization, and distillation.
Source
- Paper: *The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents*
- Authors: Darshan Tank, Baran Nama (Sentient Labs)
- arXiv: https://arxiv.org/abs/2607.22520
- Code: not yet released
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178503905