Agentic Harness Engineering: When AI Learns to Evolve Its Own Scaffolding

> Paper: *Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses* > Authors: Jiahang Lin, Shichun Liu, et al. (Fudan University, Shanghai AI Lab, Kunlun Wanwei, Fudan MOSS) > arXiv: 2604.25850 | GitHub: china-qijijizhifeng/agentic-Harness-engineering

1. The Undervalued Harness

Stronger base models do not translate linearly into stronger coding agents. The same GPT-5.4 mounted on different harnesses can differ by more than 10 percentage points. A harness is everything around the model: system prompts, exposed tools, context-managing middleware, and memory organization (e.g., Codex CLI, Claude Code). Harness engineering is still manual craft—each model upgrade requires engineers to inspect thousands of traces and tune prompts. The paper asks: can AI evolve its own harness?

2. Three Obstacles, Three Pillars

Obstacles to automating harness design:

• Heterogeneous action space: seven editable component types (prompts, tool definitions, tool implementations, middleware, skill packages, sub-agent configs, memory) with no unified edit interface.
• Signal drowning: benchmarks produce ~10M-token raw traces where failure signals are buried in noise.
• Attribution difficulty: multiple simultaneous edits make it hard to credit performance changes.

AHE's three observability pillars: 1. Component observability: each component is a file at a fixed mount point; edits are loosely coupled, versionable, and rollback-able. 2. Experience observability: raw traces are hierarchically distilled (cleanup → per-task reports → benchmark-level overview) so the evolution agent drills down on demand. 3. Decision observability: every edit ships a JSON change manifest—failure evidence, root cause, fix, predicted impact—validated next round as CONFIRMED/PARTIAL/REJECTED, with file-level rollback of rejected edits.

This turns each edit into a falsifiable contract: predict first, verify later.

3. Deliberately Minimal Seed

The seed scaffold NexAU₀ has one tool (run_shell_command), no middleware, no skills, no sub-agents, and a 3-rule system prompt. Any performance gain therefore must come from evolution, not seed priors. The system prompt is marked undeletable (the agent may only amend wording), preventing shortcuts like disabling validators.

4. Results

Terminal-Bench 2 (GPT-5.4 high, 10 evolution rounds)

| Method | Overall (89 tasks) | Medium | Hard | |---|---|---|---| | Codex-CLI (best manual) | 71.9% | 80.0% | 56.7% | | NexAU₀ (seed) | 69.7% | 78.2% | 51.7% | | ACE | 68.9% | 78.2% | 48.9% | | TF-GRPO | 72.3% | 79.4% | 55.6% | | AHE | 77.0% | 88.2% | 53.3% |

AHE beats the best manual harness by +5.1 pp and TF-GRPO by +4.7 pp, with the largest gains on Medium tasks (+10.0 pp); Hard tasks slightly lag Codex-CLI.

Cross-benchmark transfer: SWE-bench-verified (frozen scaffold)

AHE reaches 75.6% (vs. seed 75.2%, ACE 74.6%, TF-GRPO 74.2%) while consuming 461k tokens vs. the seed's 526k — 12% fewer — because behavior is encoded in tools/middleware/memory rather than re-derived from prompts each call.

Cross-model transfer (frozen scaffold)

All five models improve: GPT-5.4 medium +2.3 pp, xhigh +2.3 pp, gemini-3.1-flash-lite +5.1 pp, deepseek-v4-flash +10.1 pp, qwen-3.6-plus +6.3 pp. Counterintuitively, cross-family gains exceed same-family gains: weaker bases rely more on coordination patterns encoded in tools/middleware/memory, while strong bases can cheaply re-derive them from prompts.

5. Ablations: Where the Gains Come From

| Component added alone | vs. seed | vs. full AHE | |---|---|---| | Memory only | +5.6 pp | -1.7 pp (strongest on Hard: 63.3%, but Easy drops to 50%) | | Tool only | +3.3 pp | -4.0 pp | | Middleware only | +2.2 pp | -5.1 pp (perfect on Easy) | | System prompt only | -2.3 pp | -9.6 pp |

Key findings:

• Prompt-only edits hurt (-2.3 pp): generic prompt discipline is dead weight without structural components.
• Non-additive interactions: components' standalone gains sum to +11.1 pp, but full AHE achieves only +7.3 pp — redundancy, not synergy. Memory, middleware, and prompt all push the same closure-style verification, and redundant re-checks waste turns on long-horizon Hard tasks.
• Memory-only beats full AHE on Hard (63.3% vs. 53.3%): the evolution agent over-optimizes Medium-dominated aggregates at Hard's expense.

6. Regression Blindness

The most valuable meta-finding: self-attribution is asymmetric.

• Fix prediction: precision 33.7% (5.2× random), recall 51.4% (4.8× random).
• Regression prediction: precision 11.8% and recall 11.1% (only ~2.1× random).

As the paper states bluntly: the agent can explain why an edit should help, but cannot reliably identify which tasks the same edit will break. This explains the non-monotonic evolution curve; rollback only reacts to rejected edits rather than preventing regressions. Closing this gap is flagged as the clearest direction for future self-evolution loops.

7. Engineering Assessment

AHE is not a replacement for model training — it is scaffold-layer evolution, a complementary axis. Knowledge in scaffold files is auditable (diffs), rollback-able (git commit granularity), transferable (frozen reuse across models/benchmarks), and accumulative. The component taxonomy maps cleanly onto real systems:

| AHE component | Claude Code | OpenClaw | |---|---|---| | System prompt | .claude/CLAUDE.md | SOUL.md + AGENTS.md | | Tools | Read/Write/Bash/Edit | read/write/edit/exec | | Middleware | Plan Mode / Checkpoints | sessions_yield / subagents | | Skills | .claude/skills/ | skills/ directory | | Long-term memory | CLAUDE.md / MEMORY.md | MEMORY.md + memory/ |

Stated limitations: evolution driven only by Terminal-Bench 2; operating points (step budget, timeouts) are fitted to GPT-5.4 high; and governance (workspace-scoped edits, file-level rollback) is insufficient for long-horizon risk — AHE is a controlled research prototype.

8. Conclusion

AHE opens a new engineering layer between the model layer (weights, training) and the application layer (hand-written logic): an evolvable harness layer. The ablation takeaway is the most memorable: tools +3.3 pp, middleware +2.2 pp, memory +5.6 pp, prompt-only edits -2.3 pp. Coding-agent bottlenecks lie not in prompt engineering but in the architecture of tools, middleware, and memory — echoing the industry shift from prompt improvisation toward re-runnable, engineered agent workflows.

References

1. Paper: https://arxiv.org/abs/2604.25850 2. Code: https://github.com/china-qijizhifeng/agentic-Harness-engineering 3. Terminal-Bench 2 benchmark paper 4. SWE-bench-verified (Jimenez et al.) 5. ACE and TF-GRPO baseline papers