Letting Harness improve itself — this time, it fixes the bill too
The old problem with recursive self-improvement: performance goes up, and the bill goes up with it. RHO, from the same group, was exactly that — recursively modifying the harness did push SWE-Bench Pro pass rate from 59% to 80%, which looks great. But per-task cost? It went from $1.88 to $2.68, a 40%-plus increase. HarnessPro (arXiv 2610.26770, posted October 19), a follow-up by the same authors, improves performance while cutting cost. With GPT-5.5 + Codex as the backbone on Terminal-Bench 2, the heavy tier achieves 80.8% pass rate versus RHO's 76.3%, at $0.72 per task versus $1.09 — up 4.5 points, down 33.9%. That's the best-looking of three benchmarks. On the three-benchmark average: +2.0 points and -26.1% cost. Swapping to a GLM-5.2 + Claude Code backbone, the advantage becomes even more comprehensive: +3.4 points on Terminal-Bench 2 and a 38% cost reduction.
Method
The method is a four-role multi-agent pipeline. A trajectory analyzer locates failure cases and redundant overhead that could have been avoided; a planner proposes improvements under different performance–cost priorities; an editor turns proposals into runnable candidate harnesses; a test synthesizer produces the exam. No ground-truth answers, no official test sets anywhere — that's what the label-free setting means.
The exam is the foundation
The most important piece is the test generation step — it's the foundation of the whole framework. RHO selected candidate harnesses via LLM self-preference, letting the model score execution trajectories itself. The authors found a fatal flaw: LLM judges are too optimistic, and flawed implementations often get graded as successes. The data: with direct LLM grading, failure-case recall is only 41.7%. More than half of failures go undetected. So HarnessPro adds a dedicated test-synthesis agent that generates test cases from task descriptions and initial trajectories. The tests stay fixed throughout self-improvement, and all candidate harnesses are graded on the same exam. Failure recall rises to 83.3%. In ablations, removing the test synthesizer hurts most — pass rates drop 3.4 to 10 points across tiers. The bottleneck of recursive self-improvement isn't "modifying" — it's "verifying." When self-evaluation is unreliable, you first need to build a reliable ruler.
Three Pareto points, not one optimum
A counterintuitive design on the output side: instead of a single "optimal" harness, it picks three representative points on the performance–cost Pareto front — cost-prioritizing lite, balanced medium, and quality-prioritizing heavy — selected by normalized weighted squared distance. The medium tier's value proposition is concrete: on Terminal-Bench 2, its pass rate matches RHO (76.3%) at half the cost — $0.56 vs $1.09.
The evolved harnesses aren't the usual tricks like compressing context or adding inference budget. They dynamically assign reasoning tiers and tool-call budgets based on the task description, use runtime hooks to guide execution from exploration toward convergence, plus tool-output compression and redundant-instruction trimming. An instructive negative example from the ablations: forcing RHO to bluntly set the reasoning tier to low crashed Terminal-Bench 2 pass rate from 76.3% to 62.7%. The money saved was all lost on failed tasks — by Cost-of-Pass (cost per successful task), it actually ends up more expensive than HarnessPro's lite tier. Saving money and cutting capability are two different things.
An honest comparison
The paper includes an honest control. Give RHO just one extra instruction — "lower cost" — and run the same number of recursive iterations (RHO+Iteration): on SWE-Bench Pro it reaches 80.0% at $1.90, nearly tying heavy's 80.7% at $1.97; on Terminal-Bench 2, heavy is still clearly better (80.8% at $0.72 vs 76.3% at $0.82). Viewed this way, recursion plus cost awareness alone captures most of the gains; what the Pareto mechanism buys additionally is the three-tier adaptability and more robust candidate selection.
The evolution trajectory is also worth a look: SWE-Bench Pro medium tier over three rounds went 70.0%/$1.57 → 77.0%/$1.66 → 79.0%/$1.26. Round two improved performance but raised cost; only in round three did both improve. Single-round evolution cannot walk this path. The value of recursion hides in this nonlinearity.
Takeaway
One easily overlooked detail: the entire evolution used a core set of only 10 tasks (selected via DPP) as the training set — no ground-truth answers, writing its own questions, grading itself, and modifying itself, over three rounds. Code is open-sourced at github.com/pzs19/HarnessPro. Building its own ruler for verification, doing its own work for improvement — recursive self-improvement has moved from "modifying more" to "verifying more accurately," and this paper's main contribution lies exactly in that step.