An Empirical Study of Harness Design for Coding Agents

Forum topic · 小凯 · 2026-09-19

Summary

This arXiv paper (2609.20804) presents a component-level empirical study of coding harnesses—the scaffolding that turns autonomous coding agents' model capabilities into long-horizon software-engineering performance. Using a lightweight harness with a fixed execution loop, the authors vary three components: planning, action space, and context management. Across four models on SWE-Bench Verified and Terminal-Bench 2.1, in 176 matched settings covering five context-management strategies, four context-window budgets, and targeted ablations, key findings emerge: context management grows more valuable as context-window budgets shrink, mostly by preventing context-overflow failures; rule-based omission before LLM summarization yields the strongest efficiency, while making omitted content recoverable adds rarely-used mechanisms without accuracy gains; planning shifts from an accuracy scaffold for weak models to a cost saver for strong models; predefined tools help models with weak bash skills, whereas bash-capable models run effectively with a bash-only interface at significantly lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects, offering guidance for model- and budget-aware harness design.

Paper Overview

Field: NLP
Authors: Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang
Published: 2026-09-17
arXiv: 2609.20804

Abstract

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, the authors study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management.

Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, the study covers 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space.

Key Findings

1. Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. 2. Rule-based omission placed before LLM-based summarization provides the strongest overall efficiency among context-management strategies. Making omitted content recoverable adds a mechanism that models rarely use, without accuracy gains. 3. Planning shifts from an accuracy scaffold for weak models to a cost saver for strong models, with little change in accuracy. 4. Predefined tools improve performance for models with weaker bash abilities, while bash-capable models can operate effectively with a bash-only interface at significantly lower cost, especially on command-line-centric tasks.

Trajectory-Level Analysis

Trajectory analysis explains these effects:

Context management lengthens execution trajectories without substantially changing agent behavior.
Planning changes where trajectories terminate.
Action space changes the granularity of code edits.

These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

--- *Auto-collected on 2026-09-19*