Key points
- The problem. Foundation models iterate monthly, but the surrounding harness—system prompts, tool schemas, middleware, memory, and skill packs—is still hand-tuned by prompt engineers and DevOps, creating a structural gap between model capability and harness maturity.
- AHE's claim. Treat the entire harness as a composable whole and evolve it jointly. Prior automated optimizers touch only one surface: Reflexion/Self-Refine edit agent output; ACE and DSPy tune prompts only; Voyager and AlphaEvolve edit skill libraries. AHE is the first to evolve the full harness as one system.
- NexAU's decoupled design. The harness is split into seven orthogonal files mounted at fixed paths:
code_agent.yaml,systemprompt.md,LongTermMEMORY.md(mutable),ShortTermMEMORY.md(runtime-only),tool_descriptions/*.tool.yaml,tools/,middleware/,skills/, andsub_agents/. Each failure maps cleanly to a single component class; each logical edit corresponds to one git commit, giving file-level diffs and rollback. - Minimal seed by design. The seed harness H₀ is intentionally minimal: a single shell execution tool, no middleware, no skills, no sub-agents. Any added component must earn its place through measured rollout, keeping attribution clean.
- Three observability pillars. 1. Component observability — files instead of internal buffers, so the action space is enumerable and edits are rollable back at file granularity. 2. Experience observability — a 1000:1 distillation from raw traces (~10M tokens) into per-task analysis reports and a single benchmark-level overview, run by an Agent Debugger at concurrency 16 with a 600s per-task timeout. 3. Decision observability — every edit ships with a Change Manifest entry listing failure evidence, inferred root cause, fix, and predicted fixes/regressions. The next iteration computes the predicted-fix intersection with the actual task-level delta, producing a per-edit verdict that triggers rollback on failure. Two guardrails protect this: writes are restricted to the workspace (runs, tracer, verifier, and LLM config stay read-only; the seed system prompt cannot be deleted), and every edit must remain evidence-driven and record a falsifiable prediction.
- Terminal-Bench 2 (89 tasks), 10 iterations, ~32 hours. AHE reached 77.0% overall (+7.3 pp over the 69.7% seed), beating Codex (71.9%) and TF-GRPO (72.3%). It tied at 100% on Easy, led on Medium (88.2%), and was the only case where Hard (53.3%) lagged Codex (56.7%)—attributed to redundant verification eating into turn budget.
- Ablations. Adding long-term memory alone gave +5.6 pp and even surpassed full AHE on Hard; tool set +3.3 pp; middleware +2.2 pp. System-prompt-only substitution was -2.3 pp, showing the evolved system prompt encodes general discipline that only pays off when paired with the other three components. The three positive components sum to +11.1 pp, exceeding the +7.3 pp full-stack gain—evidence of non-additive interactions.
- SWE-bench-verified transfer (no re-evolution). 75.6% overall (django 81.0%, sympy 70.7%, sphinx-doc 70.5%), beating the seed, ACE, and TF-GRPO while using ~32% fewer tokens than ACE and ~21% fewer than TF-GRPO. ACE and TF-GRPO underperform the seed here because their playbooks/trajectories were distilled on Terminal-Bench and only add cost on a new surface.
- Cross-model transfer. The evolved workspace transfers to GPT-5.4 (medium and xhigh reasoning), qwen-3.6-plus, gemini-3.1-flash-lite-preview, and deepseek-v4-flash with gains from +5.1 to +10.1 pp; the largest gains appear on weaker backbones, which benefit most from the encoded coordination patterns.
- A shift from "hand-written prompts" to "evolvable architecture": engineers design optimization frameworks and constraints rather than tuning components line by line.
- Observability as a first principle: explicit/decoupled action space, structured distillation of raw signals, and prediction–verification loops can generalize beyond Coding Agents to any long-iteration agent system.
- Open issues: hard-task interference from redundant verification, weaker prediction of regressions than of fixes, the cost of a minimal seed, and scalability beyond the ~500-task regime.
- Paper: arXiv:2604.25850
- Code: https://github.com/china-qijizhifeng/agentic-harness-engineering
- Project blog: https://dawning-road.github.io/blog/agentic-harness-engineering
- Related work: NexAU, Terminal-Bench 2, SWE-bench-verified, ACE, TF-GRPO, DSPy, Voyager, AlphaEvolve
Experimental results
Strategic implications
Conclusion
AHE repositions the harness as automated infrastructure: auditable, rollback-able, and falsifiable. The data support a core conclusion—structural harness components transfer better than prose-level strategy, implying that Coding Agent competitiveness will move from who writes the prettiest prompt to whose harness architecture evolves faster.