An Empirical Study of Harness Design for Coding Agents

Forum topic · 小凯 · 2026-09-19

Summary

This paper presents a component-level empirical study of coding harnesses for autonomous coding agents. Using a lightweight harness with a fixed execution loop, the authors vary three components—planning, action space, and context management—across 176 matched settings involving four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, spanning five context-management strategies, four context-window budgets, and targeted ablations. Key findings: (1) context management grows increasingly valuable as context-window budgets tighten, with most gains coming from preventing context-overflow failures; (2) rule-based omission placed before LLM-based summarization yields the strongest overall efficiency, while making omitted content recoverable adds mechanisms models rarely use without accuracy gains; (3) planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little accuracy change; (4) predefined tools help models with weaker bash capability, while bash-capable models can operate effectively with a bash-only interface at notably lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects and provides guidance for model- and budget-aware harness design. arXiv: 2609.20804.

An Empirical Study of Harness Design for Coding Agents

Field: NLP
Authors: Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang
Published: 2026-09-17
arXiv: 2609.20804

Key points

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems. To enable component-level comparison, the authors fix the execution loop of a lightweight harness and vary three components: planning, action space, and context management.

Setup: Four models on SWE-Bench Verified and Terminal-Bench 2.1, covering 176 matched settings: five context-management strategies, four context-window budgets, and targeted ablations of planning and action space.

Findings:

1. Context management becomes increasingly valuable as the context-window budget tightens; most of its benefit comes from preventing context-overflow failures. 2. Placing rule-based omission before LLM-based summarization provides the strongest overall efficiency among context-management strategies. Making omitted content recoverable adds a mechanism models rarely use and brings no accuracy gains. 3. Planning shifts from an accuracy scaffold for weak models to a cost saver for strong models, with little accuracy change. 4. Predefined tools improve performance for models with weaker bash capability, while bash-capable models can run effectively using only the bash interface at significantly lower cost, especially on command-line-centric tasks.

Trajectory-level analysis: Context management lengthens execution trajectories without substantially changing agent behavior; planning changes where trajectories terminate; the action space changes the granularity of code writing.

These findings inform model- and budget-aware harness design and offer a modular framework for evaluating future harness components.

--- *Auto-collected on 2026-09-19*

An Empirical Study of Harness Design for Coding Agents

Summary

An Empirical Study of Harness Design for Coding Agents

Key points

Tags