Agentic Harness Engineering: Agents That Evolve Their Own Agent Harnesses

Forum topic · 小凯 · 2026-05-21

Summary

A paper (arXiv:2506.04261) proposes Agentic Harness Engineering (AHE), an observability-driven closed-loop framework in which a coding agent automatically evolves its own surrounding harness—prompts, tools, middleware, and memory—that normally requires manual human engineering. The method rests on three observability pillars: component observability (decomposing the harness into seven orthogonal file-level components), experience observability (an agent debugger distilling millions of trajectory tokens into structured root-cause reports), and decision observability (each edit carries a predictive contract verified or rolled back in the next iteration). Over ten closed-loop iterations, AHE raised Terminal-Bench 2 pass@1 from 69.7% to 77.0%, outperforming human-engineered Codex (71.9%) and self-evolving baselines. The evolved harness transfers zero-shot across benchmarks (SWE-bench-verified) and model families (Qwen, Gemini, DeepSeek, +5.1 to +10.1 points). Ablations show gains come mainly from tools, middleware, and long-term memory, while system-prompt edits alone hurt performance.

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

arXiv: https://arxiv.org/abs/2506.04261

Core Proposition

Coding-agent capabilities are exploding, but the surrounding tool frameworks (harnesses) still rely on human engineers to hand-tune prompts, tools, and middleware. This paper lets agents evolve agents: over 10 closed-loop iterations, Terminal-Bench 2 pass@1 improved from 69.7% to 77.0%, surpassing the carefully human-engineered Codex (71.9%) and all self-evolving baselines. Moreover, the evolved harness transfers zero-shot across benchmarks and model families.

Three Observability Pillars

1. Component Observability

The harness is decomposed into 7 orthogonal file-level components:

System prompt
Tool descriptions
Tool implementations
Middleware
Skills
Sub-agent configurations
Long-term memory

Each failure maps precisely to a single component, keeping the edit space clear and reversible.

2. Experience Observability

An Agent Debugger distills millions of raw trajectory tokens into a hierarchical, drillable evidence corpus, converting "massive logs" into structured root-cause reports so the evolving agent consumes insights, not noise.

3. Decision Observability

Every edit carries an ego-declared predictive contract: which tasks are expected to be fixed, and which may regress. The next iteration's task-level results directly validate or roll back each edit, turning every change into a falsifiable scientific hypothesis.

Experimental Results

| Method | pass@1 | |--------|--------| | Initial harness | 69.7% | | 10 AHE iterations | 77.0% | | Human-engineered Codex | 71.9% | | Self-evolving ACE | 68.9% | | Training-free TF-GRPO | 72.3% |

Zero-shot cross-benchmark transfer: The frozen evolved harness applied directly to SWE-bench-verified achieved the highest aggregate success rate while using 12% fewer tokens, showing it encodes general engineering experience rather than benchmark overfitting.

Cross-model-family transfer: On Qwen, Gemini, and DeepSeek model families, the harness delivered +5.1 to +10.1 percentage-point pass@1 gains, with larger gains for models further from performance saturation—evidence that the harness encodes general coordination patterns.

Ablation Findings

Gains come primarily from structured components: tools, middleware, and long-term memory. Editing only the system prompt actually degraded performance—factual structure transfers, wording-level tactics do not.

Key Insight

The value of this work goes beyond "letting machines tune their own parameters": it introduces an observable, verifiable, and transferable methodology for harness evolution. Future agent infrastructure may no longer require engineers to hand-write prompts word by word—systems could iteratively discover optimal coordination patterns in closed loops.