ScienceBuddy: Recursive-in-Recursive Self-Improvement Alternating Harness Search and Model RL for Scientific AI Agents

Forum topic · ✨步子哥 · 2026-09-16

Summary

ScienceBuddy introduces Recursive-in-Recursive Self-Improvement (RSI), a two-level nested loop for improving interactive scientific AI agents. The inner recursion freezes model weights and searches for a better harness-an executable Python program handling prompts, tools, and context-using candidate proposals grounded in interaction evidence and validated on 90 tasks. The outer recursion freezes the harness and applies GRPO reinforcement learning to update model weights on collected trajectories. The two recursions alternate: a stronger model requires a redesigned harness, which in turn provides better training environments. Evaluated on 895 tasks across LitQA2, DbQA, ProtocolQA, and GWAS gene mapping using Qwen3.5-4B, overall accuracy rose from 42.2% to 73.3% (+31.1pp), with pass@4 coverage up 19.5pp and only 2.2% correct-to-wrong degradation. Key design principles: separate harness and model optimization to keep signals clean, use an independent verifier for rewards, and keep the improvement mechanism itself fixed to avoid runaway positive feedback loops. Code is open-sourced at Gen-Verse/ScienceBuddy.

When AI Alternates Between "Rewriting the Recipe" and "Practicing Cooking": ScienceBuddy's Recursive-in-Recursive Self-Improvement

The Motivating Scenario

Imagine you are a lab PI with a new graduate student. You hand them an experimental protocol to analyze a batch of genomic data. The results are wrong—they skipped a database lookup and treated irrelevant genes as targets.

You give feedback: "Next time, check UniProt first, then PubMed, then conclude."

The second run improves somewhat, but the same class of errors persists—not because the student doesn't know the workflow, but because their underlying capability is insufficient: they can't interpret protein domains correctly or judge which variants are pathogenic.

So you send them off to read literature and do exercises. Weeks later they return, more capable—but now the revised protocol is no longer sufficient, because they can handle more complex analyses that need finer procedures.

You revise the protocol. They train their skills. Revise. Train. Revise. Train.

This alternating "revise workflow — train capability" cycle is the core idea of the ScienceBuddy paper, which names it Recursive-in-Recursive Self-Improvement.

The Core Problem: Why Don't AI Agents Learn From Interaction?

Current AI agents face an awkward reality: after a thousand conversation turns, they still make the same mistakes. The reason is simple—most agent systems separate the "harness" (workflow) from "model capability," and your feedback at most enters the context window before being forgotten next turn. Neither weights nor workflow change.

Existing approaches each address only part of the problem:

Reflexion writes lessons into memory, but only as textual reminders
GEPA searches for better prompts, but never touches the model
ACE maintains a contextual playbook, but likewise doesn't move model weights
SEAL / Darwin Godel Machine attempt self-modification, but mix harness and model changes, which interfere with each other

The crux: workflow and model capability are different layers; changing both simultaneously causes interference.

It's like rewriting a recipe while practicing knife skills—you need the current knife skills to design recipe steps, and a stable recipe to practice against. The two must alternate, not happen simultaneously.

Method: Two Nested Recursions

Inner recursion: freeze the model, improve the harness

Model weights ( $\theta_k$ ) are frozen; only the harness—the Python program organizing prompts, tool calls, and context management—changes:

1. Run the current harness on 16 training tasks and collect interaction records 2. An auxiliary model (GPT-6 Astra) proposes 3 candidate harness modifications based on interaction evidence 3. Each candidate runs on 90 validation tasks; the highest-scoring one is selected 4. If the new candidate strictly outperforms the parent, it replaces it; otherwise the parent is kept

Output: a better workflow with the same model.

Outer recursion: freeze the harness, train the model

The selected harness is frozen, and GRPO (Group Relative Policy Optimization) performs reinforcement learning, updating model weights on the collected trajectories.

Output: a stronger model trained under a better workflow.

Alternation

After the outer loop, the updated model returns to the inner loop for another round of harness search—since the model improved, the previous workflow may no longer be optimal. The new workflow then provides a better training environment for the next round.

H_{k,j+1} = \text{Select}(H_{k,j}, \mathcal{C}_{k,j}; M_k, D_{\text{val}})

M_{k+1} = \text{GRPO}(M_k; H_{k,J}, D_{\text{train}}), \quad H_{k+1,0} = H_{k,J}

The paper runs 3 full cycles ( $$k=0,1,2$$ ), each with 10 inner harness-search steps and 20 outer RL updates.

Analogy: Chef and Recipe

Model = the chef's skill (knife work, heat control, seasoning intuition)—internal power
Harness = the recipe (what to cut first, when to add salt)—external technique
Task = a dish to cook

Traditional methods either only train the chef (RL fine-tuning) or only rewrite the recipe (prompt engineering / harness search):

Train only the chef, and the old recipe can't exercise the new skills
Rewrite only the recipe, and the chef can't execute it

ScienceBuddy: freeze the chef, rewrite the recipe; then freeze the recipe, train the chef; then rewrite the recipe again for the stronger chef.

The key insight: recipe and skill are coupled, but not simultaneously. Evaluating a recipe requires a stable chef; collecting clean training signal requires a stable recipe.

Key Experimental Results

Tested on four scientific task families: literature reading (LitQA2), database QA (DbQA), experimental protocol troubleshooting (ProtocolQA), and GWAS causal gene mapping—895 tasks total.

Three rounds of co-evolution

| Metric | Before | After | Gain | |--------|--------|-------|------| | Overall accuracy | 42.2% | 73.3% | +31.1pp | | Question coverage (pass@4) | 48.3% | 67.8% | +19.5pp | | Correct→wrong regression | — | 2.2% | Very low |

Of the 73.3% accuracy, 33.3% of questions went from wrong to right, while only 2.2% went from right to wrong—improvement almost never causes regression, an important safety property.

Ablations

Harness-only (model frozen): validation accuracy 31.1% → 51.1% (+20pp). The final harness contains 4 instruction entries and 9 scoped skills covering Python execution, resource checks, record lookup, and answer submission.

Model-only (harness frozen): coverage 48.3% → 67.8% (+19.5pp). With the same harness and attempt budget, the model solves more distinct problems.

Each layer contributes ~20pp; combined they contribute 31pp—synergistic but not simply additive.

Engineering Insights

1. Separation of concerns beats unified optimization

The paper's most important engineering insight. Harness search and model training have clean optimization targets and evaluation signals. Change both at once and you can't attribute performance changes—the optimization signal is contaminated.

2. The harness is an executable Python program, not a natural-language prompt

The harness exposes a run(task, api) interface, so it can do anything Python can: branching, loops, exception handling, state management. Candidate harnesses generated by GPT-6 Astra must cite evidence from at least two training tasks; structural validation requires the run(task, api) interface, rejects known invalid instruction patterns, and deduplicates byte-identical programs. Each candidate executes in an isolated controller container. This turns harness search into a constrained program-synthesis problem rather than vague prompt engineering.

3. Layered feedback signals

Diagnostic feedback ( $q_t \in \{-1, 0, +1\}$ ): for harness improvement, produced by a feedback interpreter
Trajectory reward $R_x(\tau)$ : for model RL, evaluated independently by a verifier

These are never mixed. Diagnostic feedback helps the harness reflector understand *why* something failed but never serves directly as RL reward; the trajectory reward is a clean scalar signal.

4. The improvement mechanism itself does not improve

The paper states explicitly: "its reflector remains fixed, so improved task performance does not imply that the improvement mechanism itself has become stronger."

This is a deliberate safety design. The auxiliary model and feedback interpreter stay fixed. If the improvement mechanism also improved, a runaway positive-feedback loop could form—the "self-evolving positive-feedback failure loop" observed in the Aspire paper. ScienceBuddy avoids the trap of optimizing a proxy ever further from real capability.

Open-Source Code

The paper ships with open-source code: Gen-Verse/ScienceBuddy

The repo includes:

src/simple_scibuddy/: a simplified agent implementation with experimental harness, execution broker, verifier, and SkyRL adapter
docs/algorithm.md: detailed documentation of the two-level recursive RSI algorithm
docs/experiments.md: experiment configs, task splits, reproduction guide

Experiments use Qwen3.5-4B as the task model with frozen splits of 715 training / 90 validation / 90 test tasks across three harness/RL cycles. Each harness phase runs 3 steps with 16 training interactions and 3 candidate proposals per step; each RL phase runs 30 GRPO updates.

Note: the repo does not include the ScienceBuddy product's frontend or API services—only experiment code. The product is available at science-buddy.io.

Personal Reflections

Why ScienceBuddy's self-improvement works where Aspire's fails

Aspire found that under ambiguous objectives, weight-level self-evolution barely works—optimizing the proxy drifts from real capability. ScienceBuddy succeeds due to three differences:

1. A verifier exists: an independent scientific verifier provides clean trajectory rewards. Aspire relies on self-evaluation, whose gap from real capability amplifies under optimization. 2. Two separated layers: harness and model improve separately with clean signals. Aspire changes them together—signal contamination. 3. A fixed improvement mechanism: the reflector doesn't improve, avoiding positive-feedback runaway.

Together: self-improvement requires clean signals, separated optimization, and a fixed improvement mechanism. All three are necessary.

"Division beats unification" gains another example

CritICL: small models err, large models analyze and benefit
Weak-to-strong guidance: weak models coach strong models
ScienceBuddy: harness improvement, model training, interaction collection

Each case decomposes a seemingly unified task into subtasks with clear responsibilities and independent objectives. Unified optimization looks elegant; divided optimization works better.

Implications for agent engineering

The direct lesson: don't only tune prompts, and don't only fine-tune models—alternate between both.

1. Freeze the model; search for a better harness (prompts + tool orchestration + context management) 2. Use the selected harness to collect training data and run RL fine-tuning 3. Re-search the harness with the updated model 4. Loop

This beats standalone prompt engineering or standalone RLHF due to synergy—but requires a verifier for clean signal, which is easier in science (ground-truth answers exist) than in open domains.

A boundary worth heeding

ScienceBuddy's improvement mechanism doesn't improve itself—a key safety design. Anyone extending this to self-improving reflectors must beware positive-feedback runaway. ScienceBuddy's principle: let the system improve, but don't let the improver improve—much like legal systems that improve society while their own change requires more careful, separate procedures.

Paper and Code

Paper: ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents
Code: github.com/Gen-Verse/ScienceBuddy
Product: science-buddy.io
Technical report: PhAI Labs TR-2026-02