Paper Overview
- Field: Machine Learning
- Authors: Darshan Tank, Baran Nama
- Published: 2026-07-24
- arXiv: 2607.22520
- Regression: a task solved without skills but failed after skills are added.
- Residual failure: a task that fails both with and without skills.
- Regressions are substantial enough that the best performing skills outperform others primarily by regressing less, not by gaining more.
Summary
Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success. However, this metric hides an important cost: skills can also make agents worse. The paper measures both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks.
This methodology distinguishes two outcomes:
Key Findings
Three Causes of Regression
1. Skill description osmosis — a skill changes an agent's behavior merely because it exists in context, even when never invoked. 2. Grounding displacement — the skill's prescribed process overrides how the agent interprets inputs. 3. Verification displacement — the procedure suppresses checks the agent would otherwise perform on its own output.
Persistent Failures
Analysis of persistent failures reveals the same underlying pattern: existing skills over-emphasize procedural guidance—the stage that least contributes to failure—while under-weighting grounding and verification, which are the dominant sources of remaining errors.
Conclusion
After correcting for evaluation artifacts and studying agent trajectories, the authors find many regressions and persistent failures can be recovered through better grounding and verification. Procedural skills should be evaluated by decomposing their net effect into gains and regressions, rather than overall improvement alone.
---
*Auto-collected on 2026-07-28.*