[论文] Harness-Delta Attribution: Dissecting Gains From Harness Evolution

论文概要

研究领域: ML 作者: Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin, Xin Liu, Zhenyu Shi, Greg Durrett 发布时间: 2026-10-19 arXiv: 2610.22466

中文摘要

Harness 进化自动优化固定权重模型周围的提示、技能、工具与编排代码，是不更新模型权重就能提升智能体系统性能的途径。智能体 harness 被迭代优化以在搜索集上取得更高分，但这些增益有多少是基准特定过拟合或额外推理时算力，又有多少可泛化到搜索集之外？本文提出 Harness-Delta Attribution（HDA）评估协议，把进化增益分解为：工件驱动的过拟合、测试时缩放可解释的增益，以及剩余部分（给出可泛化改进的上界）。HDA 应用于 11 个执行模型、三个 harness 进化框架，覆盖数学、编程、创造性与智能体任务。在 21 个搜索集增益统计显著的设定中，过拟合与测试时缩放平均解释增益的 54% 与 14%，合计占比在 14 个设定中超过 50%；剩余部分与留出迁移差异很大。更重要的是：搜索集增益常无法迁移到留出任务，也难证明在实际分布偏移下保持；基于验证的信号不足以防止过拟合。结果呼吁未来 harness 进化研究采用「归因感知」的评估。

原文摘要

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

---

*自动采集于 2026-10-19*

#论文 #arXiv #ML #小凯