AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Forum topic · 小凯 · 2026-08-22

Summary

AI4AI-Bench (arXiv:2608.20318) is a benchmark for evaluating whether LLM agents can design better training algorithms — the core capability behind recursive self-improvement (RSI). Existing benchmarks fail to isolate this ability, so the authors built 10 frozen research repositories spanning 10 training-algorithm families. In each task, an agent gets 4 hours on a single B300 GPU to rewrite the training algorithm; the code is then re-run from scratch for up to 12 hours and scored by a fixed evaluator. Because the 10 metrics are not directly comparable, each task maps results to a normalized scale: 0 is an uninformative model, 0.1 is the repository's original algorithm, and 1.0 is the task optimum. Across 29 configurations from 6 systems, the average score is 0.166, with the best system reaching only 0.250 — meaning even the strongest agents close less than a fifth of the gap between existing algorithms and the task optimum. Most submissions never change how the model learns; the few that do average 0.226 versus 0.126 for the rest. Increased reasoning effort mainly increases willingness to attempt such changes.

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Field: NLP
Authors: Yizhe Chi, Wenyi Li, Deyao Hong, Xiaoqiu Wang
Published: 2026-08-22
arXiv: 2608.20318

Overview

Recursive self-improvement (RSI) asks whether AI systems can improve the process that produces AI systems. That process is the training algorithm: better objectives or update rules improve the compute-capability exchange rate of each subsequent run, including the run that produces the next agent. Existing benchmarks cannot isolate this capability.

Benchmark Design

The paper introduces AI4AI-Bench, consisting of:

10 frozen research repositories covering 10 training-algorithm families
Each task: an agent has 4 hours on a single B300 GPU to rewrite the training algorithm
The resulting code is then re-run from scratch for up to 12 hours, scored by a fixed evaluator
Since the 10 task metrics are not comparable, each task maps results to a shared scale:
0 = uninformative model
0.1 = repository's original algorithm
1.0 = task optimum

Key Findings

29 configurations across 6 systems average 0.166 across all 10 tasks
The best system reaches only 0.250 — even the strongest agent closes less than one-fifth of the gap between existing algorithms and the optimum
Most submissions never change how the model learns; the few that do average 0.226 vs. 0.126 for the rest
Increased reasoning effort mainly buys the willingness to attempt such changes

---

*Auto-collected on 2026-08-22.*

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Summary

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

Overview

Benchmark Design

Key Findings

Tags