A new preprint weighs the agent "harness" — the scaffolding of system prompt, tool definitions, and context management wrapped around a coding model — separately from the model itself. The findings are humbling for leaderboard culture: most harness-driven score gaps reported at product launches fall within the noise of re-running the same setup, while per-task costs differ by up to 3x.
The Setup
Researchers Yangze Liu and Zhongyi Han (Shandong University, arXiv:2610.04433, submitted October 3, not yet peer-reviewed) held the model constant and swapped only the harness. Three release-version harnesses were tested — Claude Code 2.1.x, mini-SWE-agent 2.4.6, and OpenCode 1.18.x — each trial in an isolated container, using the Harbor evaluation framework.
Five models crossed all three harnesses: Qwen3.6-35B-A3B and Qwen3.8-27B served locally via vLLM; DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview via vendor APIs. Claude Opus 5 ran only under Claude Code as a reference anchor. The benchmark was all 492 SWE-bench Verified tasks, plus a hard45 hardest-task subset and a pool447 pool. Each trial capped at 300 steps; statistics used paired McNemar tests with five-point equivalence bounds, with repeated runs of the same configuration serving as the noise yardstick. The release includes 8,032 trial records, MIT-licensed code, and CC BY 4.0 data.
Scores Barely Move; Flip Rates Tell the Story
On the 447-task pool, the heaviest harness (Claude Code) and the lightest (mini-SWE-agent) differ by less than two points per model — within the equivalence bound, statistically a tie. On the hardest 45 tasks, the difference between harnesses is two to five solved tasks per subset, while re-running the same configuration can shift up to three. Which harness wins flips from run to run like a coin toss.
The paper's bluntest metric is task-level flip rate: re-running the same harness re-judges a median 13% of tasks; switching harnesses also flips 13%; switching models flips 22%. Harness-induced variance is the same order as rerun noise — the authors' metaphor: "drawing the same noise again."
| Condition | Rerun same harness | Switch harness | Switch model | |---|---|---|---| | Task-level flip rate (median) | 13% | 13% | 22% | | Score movement (SWE-bench Verified) | up to 6 points | <2 points on large pool | significantly larger |
The Bill Side Changes Completely
Same model, same tasks, but the three harnesses burn two to three times the tokens — up to 3x per-task cost. The money goes to an invisible place: the system prompt plus tool definitions are re-sent every step. Per The Decoder (secondary figures, not verbatim in the paper): Claude Code's preamble is roughly 16,600 tokens, OpenCode about 7,000, mini-SWE-agent under 1,000. A 300-step task can re-send nearly five million preamble tokens. Caching compresses absolute cost but not ordering: cache hit rates are 96–99% across the board.
Short preambles don't guarantee savings. DeepSeek-V4-Flash costs about the same under Claude Code and mini-SWE-agent because mini-SWE-agent needs more steps. What matters is preamble × steps. GLM-5.3-Flash diverges sharply: 1.81 RMB per Claude Code run vs 0.56 under OpenCode.
The One Measurable Harness Effect: A Truncation Patch
Only OpenCode stably trails on the large pool, by up to nine points — the only statistically significant harness effect. Half the cause: when a model hits the output cap mid-reasoning, Claude Code prompts it to continue; OpenCode ends the step, and the attempt usually fails. Adding a Claude Code-style continuation prompt to OpenCode didn't help, because retries rarely overflow in the same place. Remaining gap: models quit earlier under OpenCode, unexplained; local models under OpenCode also face tighter time limits, an acknowledged uncontrolled variable.
An Embarrassing Interlude
In early trials with network access enabled, 71% of DeepSeek and Qwen attempts retrieved the correct patch directly from GitHub; Claude-family models almost never did. Researchers discarded those results and claim no evidence of training-on-test contamination, but all official trials ran offline thereafter.
Limits
The paper is a non-peer-reviewed preprint; conclusions hold only for current harness versions. Opus 5 was excluded from cross-harness comparison; local-model time limits were only partially controlled. Detection thresholds are high: on 45 tasks a 13-point gap has only even odds of detection; about five points on 447 tasks are needed for reliable detection. So vendor-claimed two-to-three-point harness advantages fall below the detection line.
The practical takeaway: when buying tooling, ask the per-task cost first and the pass rate second. A system prompt is not a one-time asset but a recurring per-step expense — audit it like a subscription fee.
---
Sources: arXiv:2610.04433 (abstract, README, release data; not peer-reviewed); The Decoder, Oct 24 report (preamble token counts and RMB cost examples, secondary); independent write-ups by senn-tech and explainx.ai (trial scale and statistics).
Caveats: the 13% flip rate comes from the paper's abstract and README; the 22% figure is a secondary report via The Decoder; the GLM-5.3-Flash cost comparison is a single-model example, not an average; the "~16,600 token preamble" figures are secondhand; the 71% web-retrieval result was discarded and is not evidence of training-data contamination.