Overview
A detailed analysis of Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses — a paper by Jiahang Lin, Shichun Liu et al. (Fudan University, Shanghai AI Lab, Kunlun Wanwei, Fudan MOSS).
- Paper: https://arxiv.org/abs/2604.25850
- Code: https://github.com/china-qijizhifeng/agentic-Harness-engineering
- Component observability: every editable component lives in a file at a fixed mount point; loosely coupled, versionable, rollback-friendly.
- Experience observability: raw trajectories (~10M tokens) are hierarchically distilled into per-task reports and benchmark-level overviews (~10K tokens each), read on demand.
- Decision observability: every edit ships a JSON change manifest (failure evidence, root cause, fix, predicted impact). Predictions are later scored CONFIRMED / PARTIAL / REJECTED; rejected edits are rolled back at file granularity.
- Memory alone: +5.6 pp; tools alone: +3.3 pp; middleware alone: +2.2 pp
- System-prompt-only edits: -2.3 pp — the only regression, showing prompt policies are dead weight without structural support
- Components are non-additive: theoretical sum +11.1 pp vs actual +7.3 pp (-3.8 pp redundancy), with redundant verification loops hurting Hard tasks
- Memory-only (63.3% on Hard) beats full AHE (53.3%) by 10 pp — the evolve agent over-optimizes Medium-dominated aggregates
- Fix predictions: precision 33.7% / recall 51.4% (5.2x / 4.8x over random)
- Regression predictions: precision 11.8% / recall 11.1% (only ~2.1x over random)
- Run a simplified AHE loop on OpenClaw's
skills/directory (single-component evolution) - Test scaffold response across other base models
- Verify how severe regression blindness is in simpler systems
- Pin down why memory-only beats full AHE on Hard tasks
- Adopt AHE's change-manifest format in OpenClaw's
MEMORY.mdupdate flow
The core question: coding-agent performance varies by more than 10 percentage points depending on the harness (system prompts, tools, middleware, memory) even with the same base model. Can AI evolve its own harness instead of relying on manual engineering after every model upgrade?
Three Structural Obstacles
1. Heterogeneous action space — editable components span seven categories (prompts, tool definitions, tool implementations, middleware, skill packs, sub-agent configs, memory) with no unified edit interface. 2. Signal drowning — a benchmark run produces ~10M tokens of raw trajectories, burying failure signals in noise. 3. Attribution difficulty — when several edits are made at once, which one caused the performance change?
Three Observability Pillars of AHE
Deliberately Minimal Seed
The seed scaffold NexAU₀ has one tool (run_shell_command), no middleware, no skills, no sub-agents, and a 3-rule system prompt. This forces every evolved component to prove its value empirically. The system prompt is protected from deletion to prevent shortcut exploits (e.g., disabling verifiers).
Key Results
Terminal-Bench 2 (GPT-5.4 high, 10 evolution rounds)
| Method | Overall (89 tasks) | Medium | Hard | |---|---|---|---| | Codex-CLI (best manual) | 71.9% | 80.0% | 56.7% | | NexAU₀ (seed) | 69.7% | 78.2% | 51.7% | | TF-GRPO (baseline) | 72.3% | 79.4% | 55.6% | | AHE | 77.0% | 88.2% | 53.3% |
AHE beats the best manual harness by +5.1 pp and the strongest self-evolution baseline by +4.7 pp, though Hard tasks slightly underperform Codex-CLI.
Cross-Benchmark Transfer (frozen scaffold → SWE-bench-verified, 500 tasks)
AHE tops the table at 75.6% while consuming 12% fewer tokens than the seed (461k vs 526k) — behavior encoded in tools/middleware/memory avoids re-derivation via prompts.
Cross-Model Transfer (same frozen scaffold, five models)
All five models improve: GPT-5.4 medium +2.3 pp, xhigh +2.3 pp, gemini-3.1-flash-lite +5.1 pp, deepseek-v4-flash +10.1 pp, qwen-3.6-plus +6.3 pp. Counterintuitively, cross-family gains exceed same-family gains: weaker models benefit more from coordination patterns encoded in structure, while stronger models can cheaply re-derive them from prompts.
Ablation Findings
Regression Blindness: A Key Meta-Finding
The evolve agent can explain benefits but cannot foresee harm:
As the paper puts it, the agent knows what it is fixing but not what it is about to break — explaining AHE's non-monotonic evolution curve.
Positioning and Limitations
AHE is not a substitute for model training; it opens an auditable, rollback-able, transferable, accumulable harness layer between model weights and application code. The paper maps its components onto Claude Code (.claude/CLAUDE.md, skills, Plan Mode) and OpenClaw (SOUL.md, skills/, subagents), arguing the approach generalizes to any file-based agent system.
Stated limitations: benchmark scope (only Terminal-Bench 2 for evolution), operating-point coupling (step budgets tuned for GPT-5.4 high), and immature governance (workspace-bounded edits insufficient for long-horizon risk). AHE should be viewed as a controlled research prototype.
Takeaway
The most memorable ablation: structure beats wording. Tool optimization +3.3 pp, middleware +2.2 pp, memory +5.6 pp — but prompt-wording edits alone -2.3 pp. The bottleneck of coding agents lies not in prompt engineering but in the architectural design of tools, middleware, and memory.