Small Models Build Agent Tools Just as Well as Frontier Models: The Harsh Truth of Harness Self-Evolution

TL;DR

> LLM agent self-evolution splits into two independent dimensions: "building tools" (harness-updating) and "using tools" (harness-benefit). Tool-building ability is uncorrelated with base model capability — Qwen3.5-9B wrote skill updates as effective as Claude Opus 4.6. But benefiting from tools is non-monotonic: weak models can't use them, mid-tier models benefit most, and strong models see reduced gains from ceiling effects. The weak-model bottleneck isn't "can't build" but "can't activate" (25% activation rate) and "can't adhere" (14% adherence rate), with long-horizon instruction adherence decaying 4x over task execution. Conclusion: spending more on the evolver is wasted; invest in the task-solving agent itself.

1. Why Building Good Tools ≠ Using Them Well

Current agent frameworks (Claude Code, AutoGPT, OpenHands) all follow the same pattern: 1. Equip the agent with external assets — prompts, skills, memory, tools (collectively the harness) 2. During execution, the agent finds "this tool isn't working well" 3. The agent modifies, adds, and optimizes these assets itself 4. Next run uses the updated assets

This is Harness Self-Evolution. The industry assumption: stronger model = better evolution, so everyone pays for Claude Opus / GPT-5 class evolvers.

The paper's answer: wrong. It decomposes evolution capability into two orthogonal dimensions:

• Harness-updating: ability to write new skills, modify tools, optimize prompts
• Harness-benefit: ability to actually benefit from updated harnesses during execution

Core finding: these capabilities are fully independent. Good builders aren't necessarily good users, and vice versa.

2. Formalization

• Base ability: M_base(f) = J_X(f, H₀)
• Pairwise evolution gain: Δ(f,e) = J_X(f, H_T^(f,e)) − M_base(f)
• Harness-updating ability (evolver view): Δ_update(e) = (1/|F*|) Σ Δ(f,e)
• Harness-benefit ability (agent view): Δ_benefit(f) = max_e Δ(f,e)

3. Finding 1: Tool-Building Is Flat Across Model Scale

Seven evolver models tested across three benchmarks:

| Evolver | SWE gain | MCP gain | SB gain | |---------|---------|---------|--------| | Claude Opus 4.6 | 7.4 pp | 3.6 pp | 2.3 pp | | Claude Sonnet 4.6 | 5.4 pp | 2.8 pp | 1.2 pp | | Claude Haiku 4.5 | 6.2 pp | 4.4 pp | 0.0 pp | | Qwen3-235B | 8.2 pp | 0.6 pp | 1.5 pp | | GPT-OSS-120B | 6.4 pp | 1.9 pp | 1.2 pp | | Qwen3-32B | 4.4 pp | 0.4 pp | 0.0 pp | | Qwen3.5-9B | 6.8 pp | 1.0 pp | 3.8 pp |

Key numbers:

• Max spread between best and worst evolver: only 3.1 pp on any benchmark
• No cross-benchmark dominant evolver: Qwen3-235B is best on SWE (8.2 pp) but worst-tier on MCP (0.6 pp)
• Qwen3.5-9B scores 3.8 pp on SkillsBench, beating Opus 4.6's 2.3 pp

Case analysis of a flink-query task showed Qwen3.5-9B's skill was structurally isomorphic to Opus 4.6's — same steps, same logic, same error handling — differing only in surface details. Writing a good skill requires understanding task structure and pattern matching, not top-tier reasoning. Likely reasons: skills are structured outputs with clear patterns; evolution is guided by execution evidence; domain knowledge can be retrieved from context.

4. Finding 2: Tool-Use Benefit Is Non-Monotonic

| Task-solving Agent | SWE base | SWE gain | MCP base | MCP gain | SB base | SB gain | |-------------------|----------|---------|----------|---------|---------|---------| | Qwen3-32B (weak) | 3.6% | 4.4 | 3.6% | 1.0 | 0.0% | 5.8 | | Qwen3-235B (mid) | 20.7% | 19.3 | 25.0% | 4.3 | 4.7% | 1.1 | | GPT-OSS-120B (mid) | 26.2% | 15.8 | 28.0% | 7.0 | 0.0% | 7.0 | | Haiku 4.5 (mid-strong) | 66.0% | 2.4 | 42.4% | 3.6 | 5.8% | 15.1 | | Sonnet 4.6 (strong) | 73.2% | 2.8 | 54.0% | 3.2 | 24.4% | 3.5 | | Opus 4.6 (strongest) | 74.2% | 2.6 | 61.0% | 3.6 | 25.6% | 5.8 |

Pattern: an inverted U. Weak models lack the ability to use upgrades; mid-tier models benefit most (enough capability to understand the harness, enough headroom to grow); strong models hit ceiling effects (Opus at 74.2% on SWE leaves only ~6% headroom).

5. Weak-Model Failure Modes: Not Building, But Using

5.1 Harness Activation Failure

Skill Load Rate (SLR):

| Model | SLR | |------|-----| | Qwen3-32B | 0.251 | | GPT-OSS-120B | 0.446 | | Opus 4.6 | 0.957 |

In a threejs task, Qwen3-32B knew it needed the skill but emitted multi-key JSON where the format gate required single-key actions — the parse failed, the skill never loaded, and the model executed unguided. Opus 4.6 correctly emitted {"load_skill": "threejs"} first. Root cause: weak models lack fine-grained format control — they know what to do but can't express it in the required format.

5.2 Harness Adherence Failure

Harness Follow Rate (HFR):

| Model | HFR | |------|-----| | Qwen3-32B | 0.142 | | GPT-OSS-120B | 0.442 | | Qwen3-235B | 0.350 | | Opus 4.6 | 0.757 |

In a pg-essay-to-audiobook task, a skill specified a fallback chain (kokoro → edge-tts → pyttsx3 → espeak → gTTS). Qwen3-32B loaded the skill, hit a FileNotFoundError on kokoro, skipped the fallback chain, and reported task_complete with no TTS available. It treated the skill as a literal script rather than a contingency program. Opus 4.6 followed the chain until finding a working tool.

5.3 Long-Horizon Adherence Decay

| Stage | Qwen3-32B | GPT-OSS-120B | Opus 4.6 | |------|-----------|--------------|----------| | After load | 0.52 | 0.67 | 0.89 | | Midpoint | 0.22 | 0.48 | 0.79 | | Final | 0.13 | 0.43 | 0.80 | | Decay | -0.39 | -0.24 | -0.09 |

Weak models' adherence decays 4x over task execution; strong models barely decay. Long-horizon instruction adherence is the core weak-model bottleneck.

6. Practical Implications: Where Should the Money Go?

| Comparison | Value | |-----------|-------| | Gap between evolvers | ≤3.1 pp | | Same-agent cross-evolver variance | ≤5.1 pp | | Gap between different agents | 36.0 pp | | Strong agent + worst evolver vs weak agent + best evolver | Strong+worst wins by 18.6–35.2 pp |

1. Invest in the task-solving agent, not the evolver. Agent capability matters 10x more than evolver capability. 2. Train harness invocation as a first-class skill. 25% vs 96% SLR shows skill loading doesn't naturally emerge — models need explicit training on when to load which skill and how to follow skill guidance. 3. Strengthen long-horizon instruction adherence. Possible approaches: explicit memory mechanisms for querying active skills, hierarchical instruction structures with checkpoints, and RL fine-tuning targeting skill adherence.

7. Relation to Existing Research

• Tool learning: using tools requires not just learning but *sustaining* — weak models learn but forget.
• In-context learning: learning from skill documents is only step one; maintaining it over hundreds of steps is the hard part.
• Long-context modeling: the problem isn't how long an input a model can process, but how long it can sustain attention/adherence across a sequence.

8. Three Counterintuitive Takeaways

1. Tool-building doesn't need a frontier model — a 9B model writes skills as good as Opus. The industry's money is in the wrong place. 2. Tool-use benefit is non-monotonic — mid-tier models gain most; strong models hit diminishing returns. 3. Weak models' bottleneck is activation and adherence, not comprehension — 75% skill-load failures, 86% adherence failures point to execution precision and long-horizon consistency.

If you build agent frameworks, ask: Can your agent load skills correctly (activation)? Does it adhere to skill guidance after loading (adherence)? Does adherence decay over long tasks (long-horizon consistency)? These matter 10x more than which frontier model you pick as evolver.

Appendix: Key Metrics

| Metric | Meaning | Weak (Qwen3-32B) | Strong (Opus 4.6) | |--------|---------|------------------|-------------------| | SLR | Skill load rate | 25.1% | 95.7% | | HFR | Harness follow rate | 14.2% | 75.7% | | LPR | Load-pass rate | 2.3% | 17.7% | | Long-horizon decay | Adherence change (start→end) | -0.39 | -0.09 |