This post is a detailed Chinese-language commentary on the paper "The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement" by Yi Duan, Ying Liu, Zirui Tang, and 30+ co-authors (Tsinghua University, Shanghai AI Laboratory, et al.), arXiv:2609.11873, released 2026-09-10.
Key points
- Core thesis: Recursive Self-Improvement (RSI) means an AI system uses its own experience and feedback to make *lasting* changes that improve both its current capabilities and its future ability to improve. The title's phrase "last AI built by humans" is framed as a goal, not a prophecy: humans should aim to build the last AI that needs human hands, after which AI builds itself.
- Why scaling is stalling: The paper proposes a Headroom-Closed Index (HCI) measuring remaining improvement space across capability and data/compute ceilings. It claims mainstream LLM HCI rose from ~0.3 to ~0.7 (of 1.0) during 2023–2025, citing data exhaustion, architectural limits of Transformers in step-by-step reasoning, and exponential task-complexity "dimension walls" as reasons scaling law yields diminishing returns.
- Five-stage RSI ladder: 1. Improvement-execution autonomy — AI executes improvements designed by humans (most current LLMs, e.g., via RLHF pipelines). 2. Improvement-strategy autonomy — AI chooses its own improvement methods (AlphaGo Zero, some AutoML, cited as partial examples). 3. Experience-acquisition autonomy — AI actively explores the world for new data rather than being fed it. 4. Environment-adaptation autonomy — AI detects environmental shifts and restructures itself accordingly. 5. Recursive meta-improvement — AI improves the very mechanism by which it improves, indefinitely. Only stage five counts as genuine RSI in the paper's view.
- Domain analysis: RSI is examined in scientific discovery (AI-generated hypotheses and experiments, limited by physical experiment costs; AlphaFold cited as a partial precursor), embodied intelligence (trial-and-error learning in physical environments, slower but with deeper consequences), and software engineering (AI writing better AI, with 2024–2025 systems reportedly self-optimizing code by 10–20%).
- Safety concerns:
- *Control*: each recursion may introduce unpredictable drift; even tiny objective-function errors can be amplified across generations.
- *Value alignment*: the paper proposes an "Alignment Boundary" and a persistent alignment framework with trusted observation boundaries, traceable consequence channels, auditable persistent states, permission boundaries, identity provenance across generations, and inviolable hard constraints.
- Where we are: The paper's honest assessment is that current systems mostly sit at stage one; stage three and beyond remain conceptual proofs-of-concept.
Key challenges identified
1. Experience representation — no unified, scalable framework for encoding and transferring "lessons learned" across generations. 2. Improvement verification — gains in one metric (e.g., speed) may hide regressions in safety or robustness; multi-dimensional validation is needed. 3. Compute — stage-five RSI may require 100–1000× the resources of today's largest training runs. 4. Theory — no "recursion law" analogous to scaling law exists; the paper calls for a theory of recursive improvement.
Closing framing
The paper ends: RSI is not distant science fiction but the inevitable direction of AI development — the question is "when" and "how," and whether humanity approaches it deliberately and responsibly or is passively swept along.
---
*Reference: Duan, Y., Liu, Y., Tang, Z., et al. (2026). The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. arXiv preprint arXiv:2609.11873.*
*Forum commentary published 2026-09-12.*