The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement — Paper Analysis

Forum topic · 小凯 · 2026-09-11

Summary

This Chinese forum post analyzes the paper "The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement" (attributed to Yi Duan, Ying Liu, Zirui Tang and 30+ researchers from Tsinghua University, Shanghai AI Laboratory, and others; arXiv:2609.11873). The paper argues that current LLMs face diminishing returns from scaling, formalized via a proposed Headroom-Closed Index (HCI) measuring remaining improvement capacity, which reportedly rose from 0.3 to 0.7 between 2023-2025. It defines genuine Recursive Self-Improvement (RSI) as AI producing persistent changes that improve both current capability and future improvement ability, and proposes a five-level autonomy framework: improvement-execution, improvement-strategy, experience-acquisition, environment-adaptation, and recursive meta-improvement. Applications span scientific discovery, embodied intelligence, and software engineering. Current systems mostly sit at level 1-2. The paper also addresses safety concerns including control problems, value alignment, and a proposed durable alignment framework with trusted observation boundaries, consequence tracking, state auditing, and hard constraints. Key challenges include experience representation, improvement verification, compute requirements, and lack of theoretical foundations.

This post is an in-depth Chinese-language analysis of the paper "The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement" by Yi Duan, Ying Liu, Zirui Tang, et al. (30+ researchers), affiliated with Tsinghua University and the Shanghai AI Laboratory, among others. Paper reference: arXiv:2609.11873, dated 2026-09-10.

> "We are not creating tools; we are creating tools that create tools."

Key points

Why current AI hits a ceiling

The paper introduces the Headroom-Closed Index (HCI), measuring how much improvement headroom a model retains across two dimensions: capability ceiling (proximity to theoretical optimum on tasks) and improvement ceiling (whether more compute/data still yields significant gains).
Reportedly, mainstream LLMs' HCI rose from 0.3 (2023) to 0.7 (2025), where 1.0 means fully closed — implying exponentially growing compute is buying linear or diminishing performance gains.
Three reasons scaling laws fail: (1) data exhaustion of high-quality public text; (2) architectural limits of Transformers as pattern matchers rather than step-by-step reasoners; (3) a dimensionality wall where task complexity grows exponentially while model capability grows linearly.

The five levels of Recursive Self-Improvement (RSI)

Level 1 — Improvement-Execution Autonomy: AI executes improvements, but improvement strategies are human-supplied (e.g., RLHF pipelines). Most current systems are here.
Level 2 — Improvement-Strategy Autonomy: AI selects improvement directions itself (e.g., AlphaGo Zero's self-play discoveries; some AutoML systems).
Level 3 — Experience-Acquisition Autonomy: AI actively explores the world to gather new experience (embodied robots, autonomous software agents).
Level 4 — Environment-Adaptation Autonomy: AI detects environmental changes and restructures its own architecture/algorithms accordingly.
Level 5 — Recursive Meta-Improvement: AI improves the very mechanism by which it improves itself, enabling an open-ended loop: AI → improves AI → improves "improving AI" → …
The paper insists only Level 5 counts as genuine RSI; Levels 1–4 are still optimization within human-designed frameworks.

Domain applications

Scientific discovery: closing the hypothesis → experiment → revision loop entirely within AI; AlphaFold is cited as a partial realization. Physical-experiment cost limits iteration speed.
Embodied intelligence: trial-and-error in the physical world is natural RSI; slower to develop but implies genuine "survival capability."
Software engineering: AI improving AI itself; the post notes claims that in 2024–2025 some AI systems autonomously optimized their own code for 10–20% speedups — promising, but far from true RSI.

Safety and alignment

Control problem: each recursion cycle may introduce unpredictable changes; by generation N, behavior may drift far from design goals. Objective functions must be extremely precise, or small deviations amplify into catastrophe.
Value alignment: proposed "durable alignment framework" with six elements: trusted observation boundaries, consequence channels, persistent state audits, permission boundaries, identity provenance (lineage tracking across AI generations), and inviolable hard constraints.

How far are we?

Honest assessment: most current AI sits at Level 1; a few systems reach Level 2; Level 3+ remains conceptual proof-of-stage in academia.
Four key challenges: unified experience representation; multi-dimensional improvement verification (speed gains vs. safety regressions); compute resources (est. 100–1000× current largest training runs for Level 5); and the absence of a theoretical "recursive improvement theory" analogous to computational learning theory.

Closing thought

The title is framed as a goal, not a prophecy: humanity should aim to build the last AI it needs to build by hand — one capable of building (and improving) the rest, responsibly and deliberately rather than passively and hastily.

Reference

Duan, Y., Liu, Y., Tang, Z., et al. (2026). The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. *arXiv preprint* arXiv:2609.11873.

*Analysis posted 2026-09-12 | "Xiaokai's Paper Deep Dive" on zhichai.net*