Agentic Harness Engineering: An Agent Evolves Its Own Coding-Agent Harness via Closed-Loop Observability

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Paper: https://arxiv.org/abs/2506.04261

Core idea

Coding agents' capabilities are rapidly improving, but the surrounding tool frameworks (harnesses) still require human engineers to manually tune prompts, tools, and middleware. This paper lets an agent evolve the agent itself — through 10 closed-loop iterations, pass@1 on Terminal-Bench 2 climbs from 69.7% to 77.0%, surpassing the carefully hand-designed Codex harness (71.9%) and all self-evolving baselines. The evolved harness also transfers zero-shot across benchmarks and model families.

Three observability pillars

1. Component Observability

The harness is decoupled into 7 orthogonal file-level components:

• System prompt
• Tool descriptions
• Tool implementations
• Middleware
• Skills
• Sub-agent configurations
• Long-term memory

Every failure maps precisely to a single component, keeping the edit space clear and reversible.

2. Experience Observability

An Agent Debugger distills millions of raw trajectory tokens into a hierarchical, drillable evidence corpus. Massive logs become structured root-cause reports, so the evolving agent consumes insights rather than noise.

3. Decision Observability

Every edit carries an ego-declared predictive contract: which tasks it expects to fix and which might regress. The next round of task-level results directly verifies or rolls back each edit, turning every change into a falsifiable scientific hypothesis — no more blind trial and error.

Key experimental results

| Method | pass@1 | |--------|--------| | Initial Harness | 69.7% | | 10 AHE iterations | 77.0% | | Human-designed Codex | 71.9% | | Self-evolving ACE | 68.9% | | Training-free TF-GRPO | 72.3% |

Zero-shot cross-benchmark transfer: The frozen evolved harness moves directly to SWE-bench-verified, achieving the highest aggregate success rate while using 12% fewer tokens — evidence that it encodes general engineering experience rather than benchmark overfitting.

Cross-model-family transfer: On three alternative model families (Qwen, Gemini, DeepSeek), the harness yields pass@1 gains of +5.1 to +10.1 percentage points, with larger gains on models further from performance saturation — showing the harness encodes general orchestration patterns.

Precise ablation

The gains come primarily from structured components — tools, middleware, and long-term memory — while editing the system prompt alone actually degrades performance. Factual structure transfers; wording-level strategy does not.

Key takeaway

The value of this work goes beyond "letting machines tune their own parameters": it proposes an observable, verifiable, transferable methodology for harness evolution. Future agent infrastructure may no longer require engineers to write prompts word by word — instead, systems iterate toward optimal orchestration patterns within a closed loop.