The Regression Tax: 5,832 Experiments Show Skill Libraries Can Break What LLM Agents Already Solved

Forum topic · ✨步子哥 · 2026-07-27

Summary

A Sentient Labs study (arXiv 2607.22520) by Darshan Tank and Baran Nama runs 5,832 paired experiments across two office-automation benchmarks (OfficeQA-Pro, SpreadsheetBench) and three model-framework stacks, testing LLM agents with and without curated skill libraries. Across conditions, skills produced 553 gains but 324 regressions—tasks the agent previously solved correctly but failed with skills—offsetting 59% of the gains. No experimental condition had zero regressions. Root-cause analysis of 81 regression cases identifies three mechanisms: skill-description osmosis (descriptions alone alter behavior without any skill being invoked), grounding displacement (skill procedures override the agent's correct reading of inputs, accounting for 72.8% of regressions), and verification displacement (skills suppress output checks the agent would otherwise perform). Failure analysis shows 90%+ of OfficeQA-Pro failures occur at the grounding stage and most SpreadsheetBench failures at verification, while existing skills overwhelmingly target the method stage. Notably, 226 of 663 formula-related SpreadsheetBench failures (34%) had correct formulas but were misgraded by the evaluator. The paper recommends reporting paired gain/regression counts, testing descriptions separately from skill bodies, and targeting skills at grounding and verification rather than method.

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

A structured summary of the Sentient Labs paper by Darshan Tank and Baran Nama (arXiv 2607.22520), based on a detailed Chinese forum write-up.

Key points

Setup: 5,832 paired experiments on two office-automation benchmarks (OfficeQA-Pro, SpreadsheetBench) across three model-framework stacks (OpenCode+MiniMax-M2.7, Codex+GPT-5.4-mini, Claude Code+Sonnet 4.6), each task run with and without a skill library, compared per-item.
Core finding: 553 gains vs. 324 regressions — skills fixed 553 previously failing tasks while breaking 324 tasks the agent previously solved, offsetting 59% of the gains. None of the 18 experimental conditions had zero regressions.
Rankings can invert: On Claude Code+Sonnet 4.6 / OfficeQA-Pro, ranking libraries by gain (12/11/10) vs. net effect gain−regression (+5/+7/+8) completely reverses the ordering. Average pass rate hides which library is genuinely safer.

Three regression mechanisms

1. Skill-Description Osmosis: Skill names and descriptions live in the system prompt even when never invoked. In one case (UID0096, correct answer 0.377), the agent passed without skills, but failed identically with all three libraries — no skill was called; the words "revised" and "customs" in descriptions alone skewed its interpretation. This defeats retrieval-based safety methods (ASSAY, GRASP, RSEA) that only evaluate skills when invoked. 2. Grounding Displacement: An invoked skill's procedure overrides the agent's correct reading of the input. 59 of 81 (72.8%) OfficeQA-Pro regressions fit this pattern — the dominant failure mode. 3. Verification Displacement: Skill procedures suppress output checks the agent would otherwise perform. Rare on OfficeQA-Pro but decisive on SpreadsheetBench.

Where failures actually live

The agent pipeline is input → grounding → method → verification → output. Existing skill libraries overwhelmingly target the method stage, but:

90%+ of OfficeQA-Pro failures are grounding errors (wrong table, year, or definition read)
SpreadsheetBench failures concentrate in verification
Residual (always-failing) tasks also stem from grounding errors that no procedural skill corrects

Evaluator blind spots

Of 663 formula-related SpreadsheetBench failures, 226 (34%) had fully correct formulas — the value-based grader simply could not evaluate structured-reference formulas. Re-computing with a real Excel engine raised pass rates by 11–49 percentage points per library (e.g., GPT-5.4-mini from 67% to 79%).

Statistical caveats

Only 5 of 18 conditions reached p<.05; only 3 survived Bonferroni correction — all Claude Code+Sonnet 4.6 on SpreadsheetBench (+43 to +47 items, p<.001). Most "improvements" elsewhere are indistinguishable from noise.

Engineering takeaways

Report paired gain/regression counts, not just net pass rates.
Evaluate skills in three conditions: no library, description-only, description+body — to detect osmosis.
Target skills at grounding and verification, not method.
Fix evaluator validation gaps — grader blind spots create false negatives that poison skill selection.

The broader lesson: "capability" is not a pass rate; it is the paired gain/regression structure. Any evaluation of a modified AI system should ask not only what it gained, but what it lost.

---

Paper: The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents Authors: Darshan Tank, Baran Nama (Sentient Labs) arXiv: 2607.22520 | Code: not yet released