The Regression Tax: Why Teaching LLM Agents New Skills Can Make Them Worse

A Chinese forum user's deep dive into the paper "The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents" (Darshan Tank, Baran Nama), based on nearly 6,000 experimental runs.

The counterintuitive finding

When researchers test whether adding a skill to an LLM agent helps, the usual method is to compare average success rates before and after. But averages lie: a skill can raise the mean success rate while *causing failures on tasks the agent previously completed*. The paper calls this cost the Regression Tax.

It's like a classroom where a new problem-solving technique raises the class average from 70 to 75 — yet one student drops from 70 to 50. The average hides the regression.

Experimental setup

• Two office-automation benchmarks, including Computer Agent Arena
• Three model harness stacks (different LLM + agent framework combinations)
• Nearly 6,000 runs, far more than typical agent papers, giving the findings statistical weight

Two failure modes

• Regression: the task succeeded *without* the skill but fails *with* it.
• Residual failure: the task fails regardless of the skill — a deeper capability or environment limitation.

Key finding: winners regress less

> Regression is so significant that the best-performing skills surpass others primarily by regressing less, not by gaining more.

A skill that enables 10 new successes but breaks 5 previously working tasks can be worse than a modest skill that adds 3 successes with zero regressions.

Three root causes of regression

1. Skill description osmosis — A skill's description changes agent behavior merely by being present in context, even when never invoked, like a formal painting in a meeting room subtly stiffening a brainstorm. Example: a "formal business writing" skill makes a simple meeting-reminder email verbose and stiff, and it even omits the meeting time, because the skill stressed formality but not completeness.

2. Grounding displacement — Prescribed operational steps override the model's direct understanding of the task, like following a GPS turn into a congested one-way street. Example: a rigid document-editing workflow (backup → open → search → replace → save) leads the model to use the wrong search term, where it would have succeeded by acting directly.

3. Verification displacement — Following a prescribed flow suppresses the model's own output checks. Like a factory inspector reassigned after automation, the model's implicit "does this result look right?" pass disappears, so errors go undetected — a major source of residual failures.

Deeper diagnosis: what skills actually get wrong

> Existing skills overemphasize procedural guidance — the least failure-prone stage — while under-supporting grounding and verification, the main sources of remaining errors.

Most failures are not because the model doesn't know *how* to act (procedural), but because it misreads the task (grounding) or fails to catch its own mistakes (verification). The cooking analogy: 90% of instruction goes to "heat oil, add vegetables, stir," but failures come from confusing sugar with salt (grounding) or never tasting the dish (verification).

Recommendations

Evaluation reform: judge skills by decomposing net effects into gains and regressions — not overall improvement alone. Measure which tasks improved, which regressed, and treat "new successes" and "avoided regressions" as separate dimensions.

Three design principles (derived from the analysis):

1. Minimal context — load skill descriptions only when needed (dynamic routing, not static injection); keep descriptions concise. 2. Preserve grounding space — frame skills as suggestions, not mandates; keep room for the model's direct task understanding; avoid over-rigid workflows. 3. Strengthen verification — build output-check checklists into skills, insert verification at key steps, and preserve the model's capacity to question its own outputs.

Good news on recoverability:

> After correcting evaluation artifacts and studying traces, many regressions and residual failures can be recovered through better grounding and verification.

Takeaway

Skills are not automatically beneficial, and more detail is not automatically better. Good skill design balances empowerment and protection: expanding what an agent can do without destroying what it already could. Skills may tax, but they shouldn't levy a hidden tax.

Reference

Darshan Tank, Baran Nama. "The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents." arXiv:2607.22520, 2026.