"Adding Feedback Summaries Made It Worse": How Meta-Harness Uses 10 Million Tokens of Diagnostic Data to Overhaul Prompt Optimization
> Paper analyzed: *Meta-Harness: End-to-End Optimization of Model Harnesses* (arXiv 2603.28052) > Authors: Yoonho Lee, Roshen Nair, Qizheng Zhang (Stanford), Kangwook Lee (KRAFTON), Omar Khattab (MIT), Chelsea Finn (Stanford) > Published: 2026-03-30 > Project page: https://yoonholee.com/meta-harness/ > Code: https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact > Analysis date: 2026-04-28, by forum analyst "Xiaokai"
1. The Real Problem: Not the Model, but the Code Wrapping It
On TerminalBench-2 (a benchmark for agentic coding), the same Claude Opus 4.6 model shows up to a 6x performance gap depending on the harness wrapping it.
A harness is not a prompt. It's the full orchestration code deciding what to retrieve, how to format it, and which state to update after each tool call. A prompt is one page; a harness is the whole book's binding, index, and page-turning logic.
If harnesses matter this much, why hand-write them? The natural idea is to apply existing text optimizers—OPRO, TextGrad, GEPA, AlphaEvolve—which iterate: try a version, get a score, generate the next version. The Meta-Harness authors argue these tools almost all fail in the harness-optimization setting, because they compress feedback too aggressively.
2. The Feedback Compression Trap: Why Summaries Hurt
An ablation comparing what information the proposer can access:
| Condition | Median accuracy | Best accuracy | |---|---|---| | Scores only | 34.6% | 41.3% | | Scores + LLM-generated summary | 34.9% | 38.7% | | Full filesystem access (Meta-Harness) | 50.0% | 56.7% |
Adding an LLM summary barely moved the median and actually dropped best accuracy from 41.3% to 38.7%.
Why? Harnesses are long-horizon programs: a design decision at step 3 (what to store in memory) may only manifest at step 47. Compressing a 47-step execution trace into a summary discards the only information that can trace a failure back to its causal design decision. Summaries aren't merely imprecise—they actively choose what to discard, and the discarded details are exactly the diagnostic ones.
It's like a doctor diagnosing from a temperature reading alone, or from another doctor's summary, instead of examining the patient, lab results, X-rays, and full medical records directly.
3. The Meta-Harness Solution: Give the Proposer the Filesystem
The core design is deliberately simple:
1. Every candidate harness evaluation stores its source code, score, and full execution trace into a filesystem directory.
2. On the next iteration, the proposer (Claude Code Opus-4.6) selectively retrieves these files via grep and cat.
3. The proposer decides what to inspect, diagnose, and change.
No preset search structure, no hardcoded parent selection, no "only the last 5 candidates." The result:
- Each evaluation step can produce up to 10,000,000 tokens of diagnostic information
- Existing text-optimization methods access at most 26,000 tokens per step
- A 385x gap—qualitative, not just quantitative
- Meta-Harness-discovered harness: 48.6% accuracy vs. ACE (best hand-designed) at 40.9%—+7.7 points—using only 11.4K context tokens vs. ACE's 50.8K (4x less).
- The key is smarter *selection*: ACE stuffs all seen examples into the prompt, while Meta-Harness learns that some examples actively interfere, so it learns selective forgetting. It strictly dominates all baselines on the accuracy-vs-context-cost Pareto frontier.
- OOD generalization: on 9 unseen datasets, Meta-Harness averages 73.1% vs. ACE's 70.2%—it learns a general classification strategy, not dataset-specific overfitting.
- Prior work found retrieval doesn't help math reasoning—but that's because naive retrieval strategies find the wrong form. Meta-Harness doesn't design retrieval strategies; it searches for them over a 535K-problem retrieval corpus.
- A single discovered harness transfers to 5 unseen models (including GPT-5.4-nano, Gemini-3.1-Flash-Lite), improving accuracy by an average of 4.7 points on 200 IMO-level problems—demonstrating cross-model transferability.
- Iterations 1–2: two candidates both fail (
evo_marker_fix→ 58.9%, -5.6pp;evo_single_confirm→ 57.8%, -6.7pp). A traditional optimizer would just try something random next. - Iteration 3: key diagnosis—the common failure factor wasn't the bugfixes but a cleanup-heavy prompt rewrite causing the agent to delete necessary state prematurely. The proposer reverts to the original prompt, keeping only the structural marker-stripping and loop-breaker fixes. Result: 63.3%, only -1.1pp. Correct diagnosis.
- Iteration 4: finds a state machine bug—verification commands reset the completion flag, trapping the agent in a "complete task → see QA checklist → run verification → flag reset → complete again" infinite loop, citing specific trace evidence from a configure-git-webserver task stuck in a 30–60 step verification spiral.
- Iteration 6: system-level optimization—"empty-command rounds waste full LLM calls when terminal output hasn't changed. Smart-waiting (polling the pane for up to 3×5 s) saves 5–15 rounds before the next LLM call." A human engineer could conceive this, but Meta-Harness found it by reading multi-hundred-step traces.
- Paper: arXiv 2603.28052, *Meta-Harness: End-to-End Optimization of Model Harnesses*
- Core innovation: agentic proposer navigates filesystem of past harness source, traces, and scores—up to 10M diagnostic tokens per step (385x more than text optimizers at 26K)
- Counterintuitive finding: scores + LLM summary (34.9% median) ≈ scores-only (34.6%), far below full filesystem access (50.0%)
- Text classification: 48.6% vs. ACE 40.9% (+7.7), 4x less context; OOD 73.1% vs. 70.2%
- Math reasoning: single harness, +4.7 points on 200 IMO-level problems across 5 unseen models
- TerminalBench-2: 76.4% with Opus 4.6 (rank 2), 37.6% with Haiku 4.5 (rank 1)
- Qualitative behavior: proposer diagnoses confounded edits, isolates causal changes, finds system-level optimizations (smart-waiting)
- Limitations: compute cost, proposer dependence, search/evaluation coupling
In practice, the proposer reads a median of 82 files and references 20+ historical candidates per step.
4. Empirical Results Across Three Domains
Online text classification: less is more
Retrieval-augmented mathematical reasoning: IMO-level problems
Agentic coding: TerminalBench-2
| Model | Best hand-designed agent | Meta-Harness | |---|---|---| | Claude Opus 4.6 | Terminus-KIRA 74.7% | 76.4% (rank 2) | | Claude Haiku 4.5 | Goose 35.5% | 37.6% (rank 1) |
The Haiku 4.5 gain is especially meaningful—weaker models are more sensitive to harness design, and Meta-Harness found a way to make it "work smart."
5. Causal Reasoning by the Proposer: One Search Trajectory
The paper's appendix documents a full search trajectory:
6. Editorial Judgment
"This isn't search—it's making the agent a doctor." Traditional text optimizers are like a nutritionist reading only the summary line of a checkup. Meta-Harness is a doctor who can flip through all records, labs, and X-rays, tracing a problem back to a drug interaction three months ago.
What "summaries made it worse" really means: LLM-generated summaries are not faithful compression—they are active distortion. Summarizers must choose what matters, and in harness optimization, the "unimportant" details are often diagnostically crucial. This is a broader warning: any pipeline that uses an LLM for information compression should ask—does the discarded material contain diagnostic signal?
Will harness engineering replace prompt engineering? Not replace—absorb. The prompt is a small part of the harness. Future LLM practitioners will ask "how do I change my harness," covering memory management, retrieval strategy, tool-call orchestration, and state machine design. Meta-Harness automates this.
Limitations: 1. Very high compute cost: a typical run evaluates ~60 harnesses over 20 iterations, each run on full trajectories of hundreds of tasks—for teams with compute budgets, not individuals. 2. Proposer dependence: with a weaker proposer, results degrade sharply—a "strong get stronger" system. 3. Search/evaluation coupling: the TerminalBench-2 experiment searches and evaluates on the same benchmark; the authors did leakage checks, but overfitting risk remains. 4. Code-space inspectability: overfitting is more checkable in code (brittle if-chains are visible), but "more checkable" isn't "automatically checkable."
Cargo cult? Partially, if teams treat it as a black-box prompt tuner without understanding *why* filesystem access beats summaries. But if they grasp the core idea—long-horizon program diagnosis requires full trace access, not compressed feedback—it's real progress.
7. Industry Implications
The deepest takeaway: the assumptions behind recent "prompt optimization" research may be wrong. OPRO, TextGrad, GEPA, and AlphaEvolve share the assumption that feedback can be compressed. That holds for short-horizon tasks; it systematically fails for harness engineering.
The lesson mirrors the big-data era: instead of building a better compression algorithm, let the optimizer query the raw data. The proposer doesn't ingest one giant prompt—it learns to navigate and retrieve from a filesystem.
If the future of LLM applications is standardized models with differentiated harnesses, then harness-search infrastructure becomes a core competitive capability—not "who has the best model," but "who has the best harness search."