Paper Overview
Research area: cs.AI / automated research / continual learning Authors: Jeremy Qin (RBC Borealis, work done during internship; ELLIS Institute Tübingen / MPI-IS), Ruizhi Deng, Mohamed Osama Ahmed, Frederick Tung, Leo Feng, Greg Mori (RBC Borealis; NeurIPS 2026 MetaAgents Workshop) arXiv: 2611.05108
Abstract
AutoResearch has shown that agents can autonomously write code, run experiments, and iteratively improve machine learning solutions. Yet most evaluations start a new agent session from scratch for each new problem, making them costly and leaving unclear whether agents can retain and apply prior insights when adapting to new tasks. We introduce Continual AutoResearch, an open-ended environment that compares persistent and reset instances of the same agent under matched task streams, data, and time budget. The environment covers four settings and nine task families (synthetic AutoML, symbolic regression, time-series forecasting, tabular AutoML) and evaluates five frontier agents (Sonnet 4.6, Opus 5, GPT-5.4, GPT-5.6-Sol, Kimi K3) on initialization quality, research efficiency, and final held-out performance. We further use memory ablations to separate transferable insights from continuous-session advantages. Across five agents, we show that persistent state improves research efficiency of all five agents, allowing them to find stronger solutions sooner. However, these gains do not consistently translate into better final solutions. Overall, our findings suggest that retaining experience and utilizing it effectively are distinct capabilities.
Why It Matters: From "Forever Newbie" to "Seasoned Researcher"
Default AutoResearch evaluation resets every session—clearing conversation history, files, and context, discarding everything learned on the previous task. This has two costs: an economic one (full trajectories are the most valuable learning data, akin to RL rollouts), and an epistemic one—the field cannot answer whether cross-task experience retention actually helps, and under what conditions. This paper flips the default and asks: does an agent get stronger if it carries its experience forward?
The answer has two layers: agents get faster, not necessarily better.
Experimental Design: Mirror Twins
- A matched task stream of ordered tasks, each with instructions, train/dev data, and a fixed wall-clock budget (default 30 minutes); final solutions scored on hidden test sets, with dev/test strictly separated to prevent reward hacking.
- Persistent execution: same agent session and workspace carry over across tasks (history, files, experiment results, execution traces).
- Reset execution: fresh session and workspace per task, everything else identical.
- Three metrics: final reward gain (outcome), first-solution quality gain (init), and time-convergence gain (normalized area under the best-so-far curve—rewarding being good *and* early).
- Metric normalization to [0,1]; p-values from 10,000 permutation tests.
- Nine task families across four settings: synthetic AutoML (shared_regime / shared_structure / shared_mechanism), symbolic regression (ten Feynman physics equations), time-series forecasting (hourly_electricity / hourly_traffic / 4_year_wind), and tabular AutoML (SELA subset + MLE-Bench Lite). Datasets are anonymized to prevent shortcut recognition.
- Five CLI agents run through native shells (Claude Code, Codex, OpenCode) with minimal scaffolding and a deliberately narrow toolset.
- Qin J., Deng R., Ahmed M. O., Tung F., Feng L., Mori G. "From AutoResearch to Continual AutoResearch: Do Agents Learn Across Tasks?" arXiv:2611.05108, 2026. https://arxiv.org/abs/2611.05108 (NeurIPS 2026 MetaAgents Workshop)
- Karpathy A. "autoresearch", 2026.
- Rank J., et al. "PostTrainBench: Benchmarking Post-Training Data Efficiency in Large Language Models." 2026.
- Chi J., Wang Z., Ma Z., Wei Z., Wei Y., Fan X., Wu L., Li F., Du Y., Zhao H., et al. "SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning." 2024.
Finding 1: Everyone Gets Faster
Time-convergence gains are positive for all five agents: Kimi K3 +21.3, Sonnet 4.6 +20.9, Opus 5 +7.5, GPT-5.6-Sol +6.1, GPT-5.4 +3.0. At 25% of budget, persistent agents lead by an average of +24.0 points; by 100%, the lead shrinks to +1.2, positive for only two of five. Persistent agents also reach their first valid solution faster (Kimi K3 saves 9.2 minutes; Sonnet 4.6, 8.6). Control experiments show the efficiency dividend comes almost entirely from starting earlier and starting stronger—not from searching better after initialization.
Finding 2: Faster ≠ Better
Final-score gains: Sonnet 4.6 +7.6 (the only solid positive), Kimi K3 +0.4, GPT-5.4 +0.2, GPT-5.6-Sol −0.8, Opus 5 −0.9. Notably, the strongest reset-state models (Opus 5, GPT-5.6-Sol) show the smallest transfer gains. Documented failure modes include: misapplying a prior signal-processing pipeline (transforming to the frequency domain when time-domain processing was needed), over-generalizing past overfitting lessons into excessive conservatism, and replacing a strong early candidate based on unreliable small-dev-set evidence—then losing the best solution because no historical-best snapshot was kept.
Memory Ablation: Session Continuity vs. Insight Transfer
A triple ablation on the energy-systems stream passes information only via agent-authored "insight cards" (claims, evidence, confidence, applicability, suggested actions, failure conditions): matched cards, mismatched (placebo) cards from unrelated streams, or none. Matched cards improved initialization for three agents but worsened their time-convergence gains; Sonnet showed the reverse pattern. Relevant experience changes behavior—starting and search paths—but being changed does not mean being improved.
Budget as a Variable
On shared_regime, tripling the budget (15/30/60 min for GPT-5.6-Sol) shrinks final and convergence gains while increasing initialization gains. Practical implication: under tight budgets, persistence is clearly worthwhile; with generous budgets, reset's cleanliness may be preferable. Budget and persistence decisions should be optimized jointly.
Positioning and Limitations
Compared to ExpeL, LifelongAgentBench, AgentCL, and SWE-Bench-CL, this work is the first to cleanly isolate experience transfer in long-horizon automated research with diverse task families and three complementary metrics. Limitations noted by the authors/forum discussion: scaffolding confounds across native CLI shells; small streams (3–5 tasks, 30 min each); memory ablation testing only insight cards on one stream; card quality unvalidated externally; and security left as an open agenda—persistent state means misleading or malicious content in one task could contaminate later ones, motivating provenance tracking and evidence-based memory revision as standard equipment.
Bottom Line
Across five frontier agents, nine task families, and mirror-twin controls with placebo ablations: experience makes everyone start faster, but only Sonnet 4.6 converts that into a final advantage. Remembering and using well are two different capabilities—the next phase for continual research agents is moving from "retaining memory" to "governing memory": tagging provenance and applicability, keeping historical-best fallbacks, and always allowing agents to question their own past.