HarnessOpt-Bench: GPT-5.6 as Architect Optimizing Other AIs' Code — Same-Family Variants Diverge Wildly, and Case Quotas Bind While Calls Are Abundant

Forum topic · 小凯 · 2026-09-03

Summary

This zhichai.net post fact-checks a video's claims about HarnessOpt-Bench, a benchmark (arXiv 2608.06301) where an optimizer LLM acts as an architect, editing the harness code of a target agent and measuring normalized gains on held-out tests. The author verifies the video's claims against the paper: the setup is real and even more rigorously sandboxed than advertised, with immutable Git commits, locked dependencies, and a trusted gateway logging every evaluation. Five models across ten configurations were tested (claude-opus-5, claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, kimi-k3, each with native CLI and opencode). Key findings: gpt-5.6-terra scores near zero (even negative on BrowseComp-Plus) while gpt-5.6-sol performs well — harness-optimization ability is an independent capability axis, not an automatic byproduct of general model skill. Claude Opus 5 leads across configurations. Optimizers exhaust ~82% of their full-case-run quota while using only ~4% of evaluation calls, showing validation budgets bind on effective sample size, not query count. Exploratory breadth of lever usage correlates positively with gains while time spent reading trajectories correlates negatively. Gains rise monotonically across five GPT generations, suggesting harness engineering is becoming a trained model capability. The post closes with honest limitations and a falsifiable 12-month prediction that optimizer capability will appear in model cards.

HarnessOpt-Bench: When GPT-5.6 Acts as Architect to Optimize Another AI's Code

A fact-checked deep dive into the HarnessOpt-Bench paper (arXiv 2608.06301v1), cross-examining a video's claims against the paper's actual numbers.

1. The Verdict Table

| Video claim | Paper reality | Verdict | |---|---|---| | AI writes code to optimize another AI | The optimizer (LLM + coding harness) edits the target agent's seed harness code, measuring normalized gains on held-out test sets | True — this is the paper's core definition | | "Strictly isolated sandbox" | The optimizer runs in an isolated sandbox; target rollouts run in ephemeral sandboxes; each candidate is an immutable Git commit; dependencies locked via lockfile; a trusted gateway logs a token ledger for every evaluation | True, and understated — the reproducibility engineering is full heavy infrastructure | | Claude Opus 5, GPT-5.6, etc. as architects | Five models, ten configurations: claude-opus-5 / claude-sonnet-5 / gpt-5.6-sol / gpt-5.6-terra / kimi-k3 × (native CLI + opencode) | True, but the video omits terra's scores (see below) | | "Two ledgers: search cost vs. deployment cost" | Paper framework: each evaluation has a cost vector; budget B caps components (100 evaluation calls per split + 4 full-case runs + a target-model token cap); the optimizer's own inference is metered but uncapped | Paraphrased terminology — "two ledgers" is video language; the structure is accurate | | "Memorized the exam, or truly understands architecture?" | The paper answers honestly: it cannot distinguish selection-induced overfitting from validation-test mismatch; it only shows that "the highest validation score seen during search is optimistic"; the sandbox is "hack-resistant, not hackproof" | True, and the paper itself admits it's unsolved | | "Real boundaries / pitfall guide" | Four boundary findings in Section 5.3 | True |

2. The Mechanism in Thirty Seconds

What gets optimized is not model weights but the harness: prompts, tool definitions, control flow, memory, and orchestration code — an executable codebase with no semantic partitioning (edit any file freely; only interfaces and a few immutable paths are locked). The invariants θ = (model, environment, verifier) cannot be touched — changing θ means changing the task, not the candidate. Data is split three ways: dev is fully disclosed (inputs + per-case results + trajectories, for diagnosis); val gives only aggregate scores (for selection); test is invisible during search and evaluated by a trusted server after the optimizer submits its final candidate. "The optimizer never sees the number it is asked to maximize" — this is the benchmark's honest foundation.

3. Four Hard Results the Video Didn't Cover

1. Night-and-day difference within the GPT-5.6 family. Normalized gains (OfficeQA / BrowseComp-Plus / Terminal-Bench / GAIA): gpt-5.6-sol under codex scores 0.49/0.03/0.12/0.49, while gpt-5.6-terra scores 0.07/−0.03/0.01/0.17 — same generation, same family; one works normally, the other nearly idles (negative gain on BrowseComp-Plus). Claude Opus 5 is the strongest across all ten configurations (0.63/0.48/0.29/0.47 under opencode), capturing two-thirds of OfficeQA's available headroom. The ability to optimize harnesses is not a natural spillover of model capability — it is an independent capability axis. Two same-family variants differ by five times the resolution band.

2. Case quotas bind; call counts are abundant. Each split allows 200 evaluation calls and 4 full-case runs. The median optimizer used only 8 calls (4%) yet burned 82% of its case budget. "Case passes bind, not evaluation calls": the real essence of a validation budget is effective sample size, not query count — evaluating each candidate requires a full agent run, which is why agent evaluation costs orders of magnitude more than text evaluation.

3. Exploration breadth > feedback depth. Across lever categories (retrieve_policy, reasoning_effort, etc.), the fraction of levers touched during search correlates positively with gains (ρ +0.34 to +0.88, same direction across four tasks), while the fraction of time spent reading evaluation trajectories correlates negatively (−0.31 to −0.64). Only 7 of 11 configurations ever requested detailed trajectories — 16 times total; per-case score summaries were enough. This adds a new variant to the "loops > X" lineage: it's not loop count, not feedback type — it's search coverage.

4. Monotonic version progress. Five generations of GPT show OfficeQA gains rising monotonically from +0.03 to +0.49 (three of four steps exceed the resolution band); Claude Opus's five generations go 0.37→0.59, non-monotonic but with a first-to-last gap beyond the band. The paper's own words: "Harness engineering is becoming a model capability, not merely infrastructure around one. The next frontier is not merely better agents, but models that reliably make agents better."

4. Connecting to the Main Thesis

A directional-sense evaluator for the self-improvement stack. In the author's six-layer stack (weights → experience → code → harness runtime → knowledge → direction sense), HarnessOpt-Bench measures the direction-sense layer itself. GPT-5.6-terra's near-zero score is an empirical sample of missing direction sense; five generations of monotonic GPT gains are longitudinal evidence that direction sense is being trained in.
Positive validation of the orchestration-tax law. The seed is a deliberately untuned ~130-line naive harness (GAIA's is even a non-functional stub) — all gain headroom lies where the static solution fails. Optimizer intelligence is tested as diagnostic repair on mature harnesses and as from-scratch construction on stubs; the paper admits it did not systematically vary seed complexity.
A precise ledger for validation-bandwidth economics. "Visible validation is optimistic": scores seen during search carry selection bias; held-out results are the real money.
A fourth form of experience carrier in place. CoE trails (unstructured trajectories) → skills (code-frozen) → FSMs (topology-frozen) → optimizer LLMs (optimization policy internalized into weights). HarnessOpt-Bench measures the capability ceiling of this fourth form; the five-generation monotonic climb suggests frontier labs are already training toward it.

5. Honest Boundaries

Anti-hacking relies on structure, not perfection (invisible test set + locked θ), but repeated dev/val feedback may still reward strategies that target the fixed evaluator — the paper suggests adding per-run jitter in future work. Overfitting vs. distribution mismatch cannot be distinguished (the "memorized exam" question the paper cannot answer). Candidates are limited to Python, one target model per task, and seed priors were not systematically controlled. The dual-use statement stands: the ability to fix agents equals the ability to attack them.

6. A Falsifiable Prediction

Within 12 months, "optimizer capability" enters model cards: release notes will start reporting HarnessOpt-Bench-style benchmarks (as SWE-bench did for coding), or Claude Code/Codex will ship "optimize your agent harness" as an official feature button. Rationale: the monotonic climb across five GPT generations shows labs are measuring this axis; once there's a benchmark there's a leaderboard; once there's a leaderboard there's productization.

---

*Verification notes: arXiv 2608.06301v1 (HTML full text captured 2026-09-04); labs.scale.com/papers/harnessopt-bench; official announcement from @ScaleAILabs on X dated 08-18; under-review OpenReview version eBSxqRO1Go. All seven video claims checked; one paraphrase-level term flagged ("two ledgers"). Model-grid numbers taken verbatim from Table 1.*