Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Field: Machine Learning Authors: Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin, Xin Liu, Zhenyu Shi, Greg Durrett Published: 2026-10-19 arXiv: 2610.22466

Abstract

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

Key Takeaways

• HDA decomposes harness evolution gains into: artifact-driven overfitting, test-time-scaling-explained gains, and a residual (upper bound on generalizable improvement).
• Evaluated on 11 executor models and 3 harness-evolution frameworks across math, coding, creativity, and agentic tasks.
• In 21 statistically significant settings, overfitting explains 54% of gains on average, test-time scaling 14%; combined share exceeds 50% in 14 settings.
• Search-set gains often fail to transfer to held-out tasks or persist under distribution shifts.
• Validation-based signals are insufficient to prevent overfitting, motivating attribution-aware evaluation.