> Paper: Meta-Harness: End-to-End Optimization of Model Harnesses > Authors: Yoonho Lee, Roshen Nair, Qizheng Zhang (Stanford), Kangwook Lee (KRAFTON), Omar Khattab (MIT), Chelsea Finn (Stanford) > arXiv: https://arxiv.org/abs/2603.28052
Key points
- A harness is the code around an LLM that determines what information is stored, retrieved, and how it is presented. The same model can vary up to 6x in performance depending on the harness.
- Counterintuitive finding: adding LLM-generated summaries to feedback *hurts*. On online text classification, scalar score + code reached 34.6% median / 41.3% best accuracy; adding summaries yielded 34.9% / 38.7%; full execution traces + code reached 50.0% median / 56.7% best — the median beats the best of the other conditions.
- Traditional text optimizers fail on harnesses because harness optimization requires long-horizon credit assignment: a decision in step 3 may only manifest at step 47. Compressing feedback into a scalar or summary destroys exactly the signal needed to trace downstream failures back to early harness decisions.
- Meta-Harness formalizes the search as
H* = argmax_H E[x~X, τ~p_M(H,x)] r(τ, x)and runs a population-based loop: evaluate harnesses, log everything (code, scores, traces) to a filesystem, and let a coding agent proposer (Claude Code + Opus) grep/cat through the history to propose new candidates. Unlike OPRO (~2K tokens/iteration) or TextGrad (~15K), Meta-Harness consumes ~10M tokens per iteration — roughly three orders of magnitude more diagnostic context. - The proposer shows non-Markovian access patterns: it reads a median of 82 files per iteration (range 69–99) and references 20+ prior candidates, citing early iterations when diagnosing later failures.
- +7.7 over ACE, +8.6 over MCE, with 4x fewer runtime context tokens.
- OOD generalization across 9 unseen datasets: 73.1% avg vs ACE's 70.2%, best on 6/9.
- Converges within ~4 evaluations to what OpenEvolve and TTT-Discover reach only after ~40, then exceeds them by 10+ points.
- Opus 4.6: 76.4% pass rate — #2 among reported agents (vs Claude Code 58.0%, Capy 75.3%, Terminus-KIRA 74.7%).
- Haiku 4.5: 37.6% — #1 (vs Claude Code 27.5%, Goose 35.5%), with larger gains on the weaker model.
- Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026). Meta-Harness: End-to-End Optimization of Model Harnesses. *arXiv preprint* arXiv:2603.28052.
- LangChain. (2026). Improving Deep Agents with Harness Engineering.
- Trivedy, V. (2026). Better Harness: A Recipe for Harness Hill-Climbing with Evals.
Benchmark results
Online text classification (GPT-OSS-120B; LawBench, Symptom2Disease, USPTO-50k)
| Harness | Law | S2D | USPTO | Avg | Context (K) | |---|---|---|---|---|---| | Zero-Shot | 12.0 | 63.2 | 7.0 | 27.4 | 0 | | Few-Shot (32) | 13.0 | 72.2 | 21.0 | 35.4 | 7.9 | | MCE | 14.0 | 83.0 | 23.0 | 40.0 | 28.5 | | ACE | 16.0 | 77.8 | 29.0 | 40.9 | 50.8 | | Meta-Harness | 14.0 | 86.8 | 45.0 | 48.6 | 11.4 |
Retrieval-augmented math reasoning (200 IMO-level problems, ≥500K-doc corpus)
The discovered harness is a four-way BM25 router (combinatorics, geometry, number theory, default), each with a distinct retrieval strategy. Averaged over 5 held-out models: 38.8 vs 34.1 no retrieval (+4.7) and 38.1 dense retrieval (+0.7), improving on all five models — avoiding the regressions dense retrieval shows on some models.
TerminalBench-2 agentic coding
Case study: the TerminalBench-2 search trajectory
| Iteration | Behavior | Result | |---|---|---| | 1–2 | Structural fixes + prompt template edits | Both regressed | | 3 | Identified confounder: template edits caused premature state deletion | Still regressed | | 4–6 | Direct fixes of diagnosed failure modes | Regressed | | 7 | Pivoted to purely additive change: environment bootstrap snapshot (working dir, file manifest, languages, package managers, memory) prepended to the initial prompt | Best candidate | | 8 | Combined two orthogonal fixes (env snapshot + marker stripping / no-tool-call loop breaker) | Further gains | | 10 | Cross-run transfer: cited results from earlier search runs | Continued improvement |
The lesson: additive changes carry zero regression risk for already-passing tasks, while modifications to fragile completion flows failed repeatedly. The agent also identified independent failure modes and combined orthogonal fixes — difficult to do in manual debugging.
Cost trade-offs
| Dimension | Text optimizers | Meta-Harness | |---|---|---| | Cost per iteration | Low (~10K tokens) | High (~10M tokens) | | Iterations to converge | Many (~40) | Few (~4) | | Runtime cost of final harness | High (ACE: 50.8K context) | Low (11.4K context) | | Human intervention | Needed (prompt tuning) | Nearly none |
A typical run evaluates ~60 harnesses over 20 iterations in a few wall-clock hours.
Limitations
1. Requires a strong coding agent as proposer (tools, code editing, verification) — only feasible with frontier models. 2. High compute cost: hours and millions of tokens per search. 3. Constrained search space: structured harness programs only, no model-architecture changes. 4. Evaluation overfitting risk when search and evaluation share a benchmark (the authors check for this). 5. Non-determinism from the proposer's non-Markovian exploration.
Takeaway: from prompt engineering to harness engineering
The paper frames a paradigm ladder: prompt engineering → prompt optimization (OPRO/TextGrad) → modular pipelines (DSPy) → harness engineering, where the full Python program around the model is optimized by an agent-driven search with system-level diagnostics. A supporting data point from LangChain: adjusting only system prompts, tools, and middleware lifted TerminalBench 2.0 from 52.8% to 66.5% (+13.7pp) with unchanged model weights. Meta-Harness automates this process: expensive to search, but the discovered harnesses run cheaper, generalize across models, and win across tasks.