SelfSearch: Reward-Free Search for Self-Improving Agents — A Feynman-Style Walkthrough

This is the third installment of a forum series on agent harnesses, told through the metaphor of a renovation crew whose experienced foreman is about to retire. The first post covered coordination (a daily stand-up that consolidates the "what's next" decision); the second covered honesty (turning plans into binding work orders). This third gate is evolution — and it's about the foreman leaving.

The question

Modern LLM agents can already inspect and modify their own instructions, tools, and execution flow — the craft of "operating on themselves" exists. What's missing is direction. Traditionally, direction comes from exam-driven evolution: propose a modification, run a downstream evaluation, keep it if the score goes up. That approach has three problems:

1. It's expensive — every modification hypothesis requires a full evaluation run (model calls, task execution, scoring). 2. Search is locked to the exam — the agent only gets good at whatever tasks the evaluation covers. 3. Most real work has no grader — debugging an internal script or reorganizing a messy knowledge base has no automatic scorer. No examiner, no selection pressure.

The modification log

SelfSearch's answer is a log. Every self-modification attempt — success or failure — produces an episode record: the reasoning behind the change, the tool actions taken, and the result. Before the next modification, the agent reads these records: why did that change fail? What did that success do right? The paper's core concept is reward-free search: a search process that needs no downstream reward signal, driven instead by records of past self-improvement. Feedback doesn't disappear — it changes form, from a scalar score to a narrative with causes and consequences, which is exactly what language models are best at learning from.

The searchable layers are three:

• Instructions — the agent's job manual
• Tools — what can be called, in what format, how errors are read
• Execution flow — the order of operations and when to stop and rethink

Over time, the log accumulates not just "which config was good" but *meta-experience*: on which layer to operate, with what strategy, and how often it works.

Results: six exams, zero losses

• Across six model–benchmark combinations, average success rate improved over the initial agent in every single one — no exceptions, no regressions.
• On Terminal-Bench 2.1 (real terminal-operation tasks), a single agent improved by up to 11.2 percentage points — genuine elevation on an already strong baseline, not just remediation.
• On SWE-bench Multilingual, success rate rose 5.0 points, and on tasks both agents could solve, execution cost dropped 38.5%. Skill up and cost down are two sides of the same coin: the log's retrospectives on "which step detoured" become straightaways in the execution flow.

One cup of coffee, matching Michelin

In the showcase experiment, SelfSearch searched for a complete harness — instructions, tools, execution flow — at a total search cost of $4.03. Installed into DeepSeek V4 Flash, the searched harness solved 82.0% of Terminal-Bench 2.1 tasks, tying Codex, the highest-scoring harness in a nine-harness public comparison.

Notably, not one cent of the $4.03 went to an examiner: no evaluation bills, no scoring labor. Compared to evaluation-guided baselines, SelfSearch is competitive on task success while being far cheaper to search, and it isn't tied to a flagship model. The strongest implication: what's truly expensive in evolution may never have been mutation — it was the invigilation.

Honest caveats

1. Without scores as a guardrail, can evolution drift? SelfSearch answers "can evolution happen without external scoring," not "where will it drift." The power source is solved; the steering wheel remains in human hands. 2. Is journaling just another form of overfitting? Lessons from task A may not transfer to task B. The six-for-six improvements suggest real generality, but "works everywhere" is not yet claimed. 3. What's the ceiling? Record-driven search extracts a model's existing abilities very thoroughly, but cannot conjure insights beyond the model's reasoning capacity. A log can turn a good hand into a master; it cannot conjure a master from nothing.

Closing: the three gates

The series' three-gate methodology, summarized: whatever relies on memory, replace with structure (coordination); whatever relies on willpower, replace with mechanism (honesty); whatever relies on external scoring, replace with self-accumulated experience (evolution). An executor has grown from a managed worker into a craftsman who polishes itself — and the tuition was $4.03.

References

• Yang, Kong & Jo. "SelfSearch: Reward-Free Search for Self-Improving Agents", arXiv:2609.37968, 2026-09-29