Meta-Harness is a system from Stanford, MIT, and KRAFTON researchers (Yoonho Lee, Omar Khattab, Chelsea Finn, et al.) that automates the engineering of harnesses—the code layer wrapping an LLM that handles prompt orchestration, retrieval, verification, and memory. Harness choice alone can cause up to 6x performance differences on the same model, yet optimizing harnesses has so far been almost entirely manual.
The problem with existing text optimizers
Prior text optimization methods (OPRO, TextGrad, AlphaEvolve, GEPA, TTT-Discover) compress feedback heavily: OPRO gets ~0.002M tokens per step, TextGrad 0.015M, GEPA 0.008M. Harness decisions have long-range dependencies—a retrieval-count choice affects hundreds of downstream samples—so compressed feedback is like seeing only an exam total without per-question work. Meta-Harness gives the proposer 10.0M tokens per step of full logs—three orders of magnitude more.
Architecture: three components, one loop
1. Filesystem as memory. All candidate sources, traces, scores, and logs live on disk in c0000/, c0001/, ... directories. No vector DB—just folders and JSON, browsable with cat/grep/diff.
2. Agentic Proposer. A coding agent reads history freely and proposes new harnesses. It must write a hypothesis before editing—explaining why, not just what.
3. External evaluator. Verification and scoring run entirely outside the proposer. AI proposes; objective metrics judge.
The full visible history removes path dependence: the proposer can revive ideas from 20 rounds ago rather than chasing recent failures.
Results
- Text classification (9 OOD benchmarks): The discovered Label-Primed Query strategy beats the best handcrafted ACE system by 7.7 points using only 1/4 the context tokens. It primes the full label space, adds one coverage example per class, then adds query-anchored contrastive pairs—same neighborhood, different labels. No human engineer designed this. Eight Pareto-optimal candidates span the accuracy–cost frontier.
- Math retrieval: From a 535K-problem database, Meta-Harness found a four-route BM25 harness (combinatorics: 20→dedupe to 8→rerank→top 3; geometry: 1 fixed + 2 nearest; number theory: 12→rerank→3). Evaluated on only 88 problems during search, it transfers to 5 held-out models with +4.7 points average.
- TerminalBench-2 agent coding: Starting from Terminus-KIRA, the system found that injecting an environment snapshot (cwd, installed languages, package managers, memory) into the initial prompt saves 2–4 exploration turns, improving 7 of 89 tasks—all dependency-heavy toolchain tasks.
- Skill text is the #1 lever; run 3–5 short evolution rounds to tune it first.
- Weak-ish baselines and 50–100 hard samples beat saturated baselines and big datasets.
- Keep logs boring: JSON, consistent naming, grep/regex accessible; add a tiny CLI for Pareto front and diff viewing.
- Light validation before full evaluation (import + instantiate + small-sample call).
- Evaluation always outside the proposer.
Practical lessons (from Appendix D)
Relation to DSPy
DSPy is a domain-specific compiler optimizing prompts and few-shot examples within its framework; Meta-Harness optimizes arbitrary executable code, including DSPy programs, framework-free.
Takeaways
1. Memory/completeness of context may be the first-order bottleneck of AI optimization systems, not compute. 2. Plain filesystem + JSON + natural-language skills—a deliberately simple stack—wins when paired with capable coding agents and large context windows. 3. Harness optimization may matter more than finetuning for most teams: it's cheaper, safer, and is the day-to-day lever.
Paper info: *Meta-Harness: End-to-End Optimization of Model Harnesses*, arXiv 2603.28052v1 (2026-03-30), cs.AI. Code: github.com/stanford-iris-lab/meta-harness-tbench2-artifact; open-source implementation: github.com/SuperagenticAI/metaharness (PyPI: superagentic-metaharness v0.4.0); demo: yoonholee.com/meta-harness/.