Key points
A Fudan team's Agentic Harness Engineering (AHE) addresses a structural gap: base models iterate monthly, but the surrounding harness (system prompts, tool sets, middleware, memory, skills) still depends on hand-written engineering. AHE asks how to let an evolution agent jointly and reliably evolve all editable harness components.
Why existing methods fall short
- Reflexion / Self-Refine: only revise agent output, not harness structure
- ACE / DSPy: optimize prompts only
- Voyager / AlphaEvolve: edit skill libraries but not the full harness
- AHE's breakthrough: treat the complete harness as a composable whole to be jointly evolved
- SWE-bench-verified (no re-evolution): AHE hits 75.6% overall vs. 75.2% seed, with 32% fewer tokens than ACE and 21% fewer than TF-GRPO; ACE/TF-GRPO both fall below seed due to distribution mismatch.
- Cross-model (GPT-5.4, qwen-3.6-plus, gemini-3.1, deepseek-v4): +5.1 to +10.1pp, with largest gains on weaker models — structured harness support helps less capable models more.
- Component interference on Hard tasks: memory-only beats full AHE on Hard; the evolution lacks interaction awareness between components.
- Regression prediction is weaker than fix prediction; the evolution curve is non-monotonic.
- Seed minimalism raises early exploration cost — a trade-off vs. attribution purity.
- Scalability: 89/500-task benchmarks; cost of rollouts and distillation at thousands of tasks is untested.
- AHE paper: arXiv:2604.25850
- Code: https://github.com/china-qijizhifeng/agentic-harness-engineering
- Blog: https://dawning-road.github.io/blog/agentic-harness-engineering
- Frameworks/benchmarks: NexAU, Terminal-Bench 2, SWE-bench-verified, ACE, TF-GRPO, DSPy, Voyager, AlphaEvolve
Three structural obstacles
1. Heterogeneous action space — components span prompts, tools, middleware, memory, etc. 2. Signal drowning — one benchmark run produces raw traces up to ~10M tokens 3. Attribution difficulty — did an edit help, or did the task mix change?
Three observability pillars
1. Component observability: the harness is decoupled into seven orthogonal components exposed as explicit files in a workspace (yaml config, Jinja system prompt, long/short-term memory, tool YAMLs, Python tools/middleware, skills, sub-agents). Each edit maps to a git commit with file-level diff/rollback. 2. Experience observability: layered distillation (~1000:1 compression) — raw traces → per-task root-cause reports → single benchmark-level overview. The Debugger runs at concurrency 16, 600s per-task timeout. 3. Decision observability: every edit ships a Change Manifest (failure evidence, inferred cause, fix, predicted fixes and regressions). Next iteration computes the intersection between predictions and actual deltas; unmet predictions trigger rollback — evolution becomes a falsifiable contract, not trial-and-error.
Minimal seed design
The seed harness H₀ is deliberately minimal: one shell execution tool, no middleware, skills, or sub-agents. This keeps attribution clean — every added component must earn its place through measured rollouts.
Results
Terminal-Bench 2 (10 iterations, ~32h):
| Method | All (89) | Medium (55) | Hard (30) | |---|---|---|---| | Codex | 71.9% | 80.0% | 56.7% | | NexAU₀ (seed) | 69.7% | 78.2% | 51.7% | | TF-GRPO | 72.3% | 79.4% | 55.6% | | AHE | 77.0% | 88.2% | 53.3% |
Ablations: long-term memory +5.6pp, tools +3.3pp, middleware +2.2pp (sum +11.1pp > full AHE's +7.3pp, indicating non-additive interactions). System prompt alone: −2.3pp, showing AHE's prompt encodes discipline that only works with the other components in place.
Transfer:
Limitations
Strategic takeaway
AHE's real contribution is proving the harness can be an evolvable architecture, akin to the shift from hand-written assembly to compiler optimization. Structural components (tools, middleware, memory) transfer better than prose-level strategy (system prompts), suggesting coding agent competitiveness will shift from prompt craftsmanship to automatable harness architecture.