Key points
- One-line summary: Stanford IRIS Lab research shows that when an AI optimizer can read complete failure "case files" (up to 10 million tokens), it outperforms human engineers at optimizing its own harness. Result: the small Haiku model reaches top performance, and the same model can swing from 26% to 59% accuracy depending solely on its harness.
- Iterations 1–2: Structural fixes bundled with prompt-template changes regressed sharply (from a 64.4% baseline). Hypotheses involved
__CMDEND__marker leakage causing tool-call loops, and double-confirmation mechanisms causing 15–40+ wasted verification steps. - Iteration 3: The proposer explicitly inferred that regressions were caused by the prompt changes, not the structural bugfixes — "the structural bugfixes were confounded with harmful prompt changes" — and isolated the verified fixes (marker stripping, loop breaker).
- Iterations 4–6: Further targeted fixes to completion logic and system-level polling still regressed, teaching the proposer that prompt/completion-flow changes are high-risk.
- Iteration 7 (winner): Rather than modifying the control loop, the proposer added an environment snapshot before the first LLM turn —
_gather_env_snapshot()collects the working directory,/applisting, available languages (python, gcc, node, java, rustc, go), package managers, and memory, injected as an[Environment Snapshot]block. This eliminated the 2–4 exploration rounds agents normally spend discovering their environment. - Text classification (GPT-OSS-120B; LawBench-215, Symptom2Disease-22, USPTO-50k): after 20 evolution rounds / 40 candidates, the best harness hit 48.6% accuracy, +7.7 points over the ACE SOTA, using only 45.5K context tokens vs. ACE's 203K.
- Math reasoning: the search evolved a retrieval-augmented reasoning strategy over a 500K-question corpus (8 open datasets), tuned on 250 search questions and evaluated on 200 IMO-level problems plus 5 models never seen during search. One retrieved harness transferred across all 5 models, averaging +4.7 points (34.1% → 38.8%) — evidence the discovered strategies generalize rather than overfit.
- High priority: full execution-trace storage (low difficulty), counterfactual diagnosis capability (medium)
- Medium priority: environment snapshot injection, harness version control
- Lower priority: full automated harness search
- Practical steps: enhanced logging of complete traces, retrieval tools (grep/cat) over history, automatic environment-awareness, automated failure analysis, and A/B testing frameworks for harness configurations
- Paper: arXiv:2603.28052
- Project page: yoonholee.com/meta-harness/
- TerminalBench-2 harness code: github.com/stanford-iris-lab/meta-harness-tbench2-artifact
- Author: Yoonho Lee on X
A counterintuitive discovery
In February 2026, engineer Can Bölük changed only the edit format — nothing else — across 15 LLMs. Coding performance improved by 5–14 percentage points and output tokens dropped ~20%. Most striking: GPT-4 Turbo's accuracy jumped from 26% to 59% just by switching edit format. The only variable was the harness, not parameters, data, or training.
What is a harness?
If the LLM is the brain, the harness is the body that lets it work. It includes system prompts, tool definitions, retry logic, context management, sub-agent orchestration, and lifecycle hooks. Agent = Model + Harness. Harness engineering today is manual and iterative — and many failure modes are not human-diagnosable from scalar scores or compressed summaries.
The core breakthrough: 400× more information
Prior text-optimization methods compress feedback heavily:
| Method | Context visible per step | |---|---| | Self-Refine | ~1,000 tokens | | OPRO | ~2,000 tokens | | TextGrad, AlphaEvolve, GEPA | ~8,000–26,000 tokens | | Meta-Harness | up to 10,000,000 tokens |
Meta-Harness gives the proposer a full file system containing all historical harness source code, execution traces, command logs, errors, timeout behavior, and scores. The proposer browses it with standard tools like grep and cat — becoming an agent that retrieves, reads history, and edits code rather than reasoning over a fixed prompt.
The search loop: (1) proposer reads the file system's history, (2) analyzes failures and causes, (3) rewrites harness code, (4) new harness runs tests with results written back, (5) repeat. On a 19-task subset, accuracy rose from a 28.5% Terminus-KIRA baseline to 46.5% by iteration 7, driven by counterfactual diagnosis of concrete traces.
Results on TerminalBench-2
TerminalBench-2 contains 89 Docker-ized tasks spanning code translation, distributed ML configuration, systems programming, bioinformatics, and cryptanalysis, scored binary and averaged over 5 runs.
| Model | Harness | Success rate | Rank | |---|---|---|---| | Claude Opus 4.6 | Meta-Harness | 76.4% | #2 among Opus agents | | Claude Haiku 4.5 | Meta-Harness | 37.6% | #1 among Haiku agents |
Haiku is the lightest model in the Claude family — Meta-Harness shows small-model ceilings can be substantially raised through harness optimization.
How the proposer "thinks"
Beyond code
Why it matters
1. Models are becoming commodities; the harness decides success. Frontier models now perform similarly — the same model with different harnesses can differ by 2× in accuracy. 2. Optimizers need eyes. Like a doctor diagnosing with full medical records versus a single "fever" note, a proposer with complete traces can trace failures to specific harness decisions; with only a score, it can only guess. 3. Good news for small models. If optimized harnesses close the gap between small and large models, deployment costs fall sharply — a major win for edge computing, real-time applications, and cost-sensitive scenarios.
Implications for agent frameworks
Conclusion
Meta-Harness marks a paradigm shift: from optimizing models to optimizing harnesses, from manual tuning to automated search, from compressed feedback to complete trajectories. Harness optimization is becoming the next battlefield of agent engineering.