Harness or Model? Isolating the Harness Effect in Agentic Coding (arXiv 2609.11987)

Forum topic · 小凯 · 2026-09-15

Summary

This paper tests the common assumption that vendor-native harnesses (tools, prompts, and control flow coupled to language models) outperform neutral harnesses in agentic coding. Using a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks, the authors ran paired same-model comparisons: claude-opus-4-8 under claude-agent-sdk vs deepagents, and gpt-5.5 under the openai-codex SDK vs deepagents, with gemini-3.5-flash and deepseek-v3.2 as side cells. Of 800 planned runs, 792 were graded by an isolated oracle. Neither contrast showed an average harness advantage: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%). Opus results mixed opposite strata: the native harness trailed by 9.0 pp on repository tasks but led by 23.7 pp on contest tasks (p=0.003). Cost re-billing suggests neutral harnesses cost 1.2-1.6x more per solved task, though missing telemetry leaves billing order unresolved. Orchestrator, oracle, and analysis code are open-sourced; tasks remain private.

Paper Overview

Field: NLP / Agentic Coding Author: Mohsen Arjmandi Published: 2026-09-15 arXiv: 2609.11987

Abstract (translated from Chinese summary)

An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. This paper measures that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks.

Experimental Setup

The same 80 tasks ran under claude-agent-sdk vs deepagents on claude-opus-4-8, and under the openai-codex SDK vs deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells.
792 of 800 planned runs were graded by an isolated judging oracle.

Key Findings

Neither contrast resolves an average advantage for either harness:
Opus 4.8: -1.25 pp (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5])
GPT-5.5: +1.25 pp (55.6% vs 54.4%, CI [-4.4, +6.9])
Opus's overall average mixes opposite strata: the native harness trails by 9.0 pp on 61 repository tasks but leads by 23.7 pp on 19 contest tasks (label-permutation p=0.003). This split was chosen after seeing the data and awaits a designed replication.
Correctness and completion are separable: of 81 runs canceled at the clock limit, 22 had already produced test-passing patches.
Re-billing raw per-turn usage at frozen catalog prices, the neutral harness costs 1.3-1.6x more per solved task on Opus 4.8 and 1.2x more on GPT-5.5. These are observational-usage estimates; 58 runs on Anthropic accounts left no usage records, which could move the Opus ratio anywhere from 0.7 to 2.3, so the billing ordering remains unresolved.

This revision corrects an August 2026 manuscript whose cost figures were based on a usage-semantics defect in the authors' own telemetry (Section 5.1).

The orchestrator, grading oracle, re-analysis code, and derived aggregate data are open-sourced. The tasks themselves remain private.