Automated Discovery Has No Universally Superior Harness

Paper Overview

• Field: NLP
• Authors: Akshat Gupta, Jermaine Lei, Alexander Lu, Gopala Anumanchipalli, Leshem Choshen
• Published: 2026-07-20
• arXiv: 2607.18235
• Categories: cs.CL, cs.AI

Summary

Autonomous discovery systems such as OpenEvolve and TTT-Discover are commonly used as general-purpose harnesses. In practice, however, these are composite systems that combine multiple design choices about the archive, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing frameworks are typically compared using too few independent trials, making it impossible to distinguish key methodological improvements from run-to-run variance.

The authors systematically decompose OpenEvolve-style evolutionary search and TTT-Discover search frameworks into their constituent components, and evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts with statistically analyzed repeated trials.

Key Findings

• Discovery harnesses exhibit a generalization problem: no fixed harness is consistently superior across all evaluated model-problem pairs.
• Variants of OpenEvolve often underperform simpler alternatives.
• Harness selection should therefore be treated as a hyperparameter rather than a universal recipe, tuned to the specific problem and the underlying model.
• The paper proposes an adaptive-allocation approach that outperforms fixed harness selection.

Original Abstract (Condensed)

> Autonomous discovery systems are often used as general-purpose harnesses, but no fixed harness is reliably superior across evaluated model-problem pairs. We systematically evaluate 30 budget-matched harnesses and propose an adaptive-allocation approach outperforming fixed harness selection.

--- *Auto-collected on 2026-07-22*