AI4AI-Bench: Benchmarking LLM Agents on Algorithmic Design for Recursive Self-Improvement

Forum topic · 小凯 · 2026-08-22

Summary

This paper introduces AI4AI-Bench, a benchmark for evaluating whether large language model (LLM) agents can engage in recursive self-improvement (RSI) by rewriting AI training algorithms. The benchmark comprises 10 frozen research repositories spanning 10 training algorithm families. In each task, an agent has 4 hours on a single B300 GPU to rewrite a training algorithm; the resulting code is then re-run from scratch for up to 12 hours and scored by a fixed evaluator. Scores are normalized per task on a scale where 0 is an uninformed model, 0.1 is the repository's original algorithm, and 1.0 is the task-optimal solution. Across 6 systems and 29 configurations tested on all 10 tasks, the average score was 0.166, with the best system reaching only 0.250—closing less than one-fifth of the gap between existing algorithms and the optimum. Most submissions never altered how models learn; submissions that did averaged 0.226 versus 0.126 for the rest. Increased reasoning effort mostly bought willingness to attempt change rather than improved results. Authors are Yizhe Chi, Wenyi Li, Deyao Hong, and Xiaoqiu Wang.

Paper Overview

Field: NLP
Authors: Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang
Released: 2026-08-22
arXiv: 2608.20318

Abstract Summary

Recursive Self-Improvement (RSI) investigates whether AI systems can improve the process that produces AI systems. Specifically, this process involves training algorithms: better objectives or update rules improve the compute–capability exchange rate of each subsequent run, including runs that produce the next agent. Existing benchmarks cannot isolate this capability.

This paper proposes AI4AI-Bench, which contains 10 frozen research repositories covering 10 training algorithm families. In each task, the agent has 4 hours on a single B300 GPU to rewrite the training algorithm; its code is then re-run from scratch for up to 12 hours and scored by a fixed evaluator. Because the 10 metrics are not directly comparable, each task is mapped onto a unified scale: 0 corresponds to an uninformed model, 0.1 to the repository's original algorithm, and 1.0 to the task-optimal solution.

Key Findings

Scope: 6 systems and 29 configurations were evaluated across all 10 tasks.
Overall performance: The average score across all submissions was 0.166, with the best system reaching 0.250.
Gap to optimum: Even the strongest system closed less than one-fifth of the distance between existing algorithms and the task-optimal solution.
Behavioral observation: The majority of submissions never changed how the model learns. The small subset that did make changes averaged 0.226, compared to 0.126 for the rest.
Reasoning effort: Additional reasoning effort mostly increased the agent's willingness to attempt algorithmic changes, rather than translating into higher final scores.

Implications

AI4AI-Bench exposes a substantial gap between current LLM-based agent capabilities and the demands of recursive self-improvement through algorithmic redesign. The benchmark suggests that while LLM agents can occasionally propose meaningful modifications to training algorithms, most systems default to leaving established pipelines untouched, limiting realized gains.