AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Forum topic · 小凯 · 2026-08-22

Summary

AI4AI-Bench is a new benchmark for evaluating whether LLM agents can design training algorithms for recursive self-improvement (RSI), the question of whether AI systems can improve the process that produces AI systems. The benchmark includes 10 frozen research repositories covering 10 families of training algorithms. In each task, an agent has 4 hours to rewrite the training algorithm on a single B300 GPU; the code is then re-run from scratch for up to 12 hours and scored by a fixed evaluator. Because the 10 metrics are incomparable, each task maps results onto a common scale: 0 for an uninformative model, 0.1 for the repository's original algorithm, and 1.0 for the task optimum. Across 29 configurations from 6 systems evaluated on all 10 tasks, the average score was 0.166, with the best system reaching 0.250—meaning even the strongest systems closed less than one-fifth of the gap between existing algorithms and the optimum. Most submissions never changed how the model learned; the few that did averaged 0.226 versus 0.126 for the rest. Increased reasoning effort mainly increased the willingness to attempt such changes. Paper: arXiv 2608.20318.

Paper Overview

Field: NLP Authors: Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang Published: 2026-08-22 arXiv: 2608.20318

Abstract (translated)

Recursive self-improvement (RSI) asks whether AI systems can improve the process that produces AI systems. That process is the training algorithm: better objectives or update rules improve the compute-capability exchange rate of every subsequent run, including the run that produces the next agent. Existing benchmarks cannot isolate this capability.

This paper introduces AI4AI-Bench, consisting of 10 frozen research repositories spanning 10 families of training algorithms. In each task, the agent has 4 hours to rewrite the training algorithm on a single B300 GPU; the resulting code is then re-run from scratch for up to 12 hours and scored by a fixed evaluator. Because the 10 metrics are not comparable, each task maps results onto a unified scale: 0 for an uninformative model, 0.1 for the repository's original algorithm, and 1.0 for the task optimum.

Key Findings

29 configurations from 6 systems were evaluated across all 10 tasks.
The average score was 0.166; the best system reached 0.250 — even the strongest closed less than one-fifth of the distance between existing algorithms and the optimum.
Most submissions never changed how the model learned; the few that did averaged 0.226 versus 0.126 for the rest.
More reasoning effort mainly bought the willingness to attempt such changes.

--- *Auto-collected on 2026-08-22*