Paper Overview
- Field: NLP
- Author: Mohsen Arjmandi
- Published: 2026-09-15
- arXiv: 2609.11987
- No average harness advantage: Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task bootstrap 95% CI [-10.0, +7.5]); +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]).
- Stratified reversal for Opus: The vendor harness trailed by 9.0 pp on 61 repository tasks but led by 23.7 pp on 19 contest tasks (label-permutation p=0.003). This split was chosen after seeing the data and awaits a designed replication.
- Correctness vs completion diverge: Of 81 runs canceled at a clock cap, 22 had already produced test-passing patches.
- Cost estimates: Re-priced from raw per-turn usage at frozen catalog prices, the neutral harness costs 1.3-1.6x more per solved task on Opus 4.8 and 1.2x more on GPT-5.5. These are observational estimates; 58 runs on an Anthropic account left no usage records, so the billing ranking remains unresolved (Opus ratio could shift between 0.7 and 2.3).
- Revision note: This revision corrects an August 2026 manuscript whose cost figures were based on a usage-semantics defect in the authors' own telemetry (Section 5.1).
Abstract (English translation)
An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks.
The authors measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle.
Key findings
--- *Auto-collected on 2026-09-15*