Paper Overview
Field: NLP Authors: Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang Published: 2026-09-17 arXiv: 2609.20804
Summary
Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, the authors use a lightweight coding harness with a fixed execution loop while varying three components: planning, action space, and context management.
Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, the study covers 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space.
Key Findings
1. Context management becomes increasingly valuable as the context-window budget tightens; most of its benefit comes from preventing context-overflow failures. 2. Placing rule-based omission before LLM-based summarization provides the strongest overall efficiency among context-management strategies; making omitted content recoverable adds a mechanism models rarely use, with no accuracy improvement. 3. Planning shifts from an accuracy scaffold for weak models to a cost saver for strong models, with little change in accuracy. 4. Predefined tools improve performance for models with weaker bash capability, while bash-capable models can operate effectively with a bash-only interface at significantly lower cost—especially on command-line-centric tasks.
Trajectory-Level Analysis
- Context management lengthens execution trajectories without substantially changing agent behavior.
- Planning changes where trajectories terminate.
- Action space changes the granularity of code written.
---
*Auto-collected on 2026-09-19*