"The Last AI Built by Humans": A 75-Page RSI Survey Review and Cross-Check — Only 3 L4 Systems Industry-Wide

Forum topic · 小凯 · 2026-09-16

Summary

A Chinese tech forum post cross-references a 75-page arXiv survey (2609.11873) on recursive self-improvement (RSI) in AI, authored by Xuanhe Zhou's team (Shanghai Jiao Tong University / Theseus Labs) with contributors from Tsinghua, ByteDance, and Shanghai AI Lab. The survey introduces the Headroom-Closed Index (HCI), a unified metric normalizing benchmark progress from first-year 90th-percentile baselines, showing 2026 gaps concentrated in long-horizon, stateful tasks (software engineering 52.6, tool agents 39.9 vs. cybersecurity agents 91.9). It proposes a five-level autonomy framework based on which improvement decisions AI controls, and its industry appendix classifies 144 systems: roughly 120 at L1-L3, only 3 at L4 (environment adaptation), and 11 at L5 (recursive inheritance). The author maps survey levels onto their own six-layer self-improvement stack, discusses three challenges (safety inheritance, autonomous attribution, reliable verification), highlights Theseus's co-evolution experiments showing +18.65 to +51.6pp gains from environment restructuring alone, and predicts L4 verification bandwidth will become the field's key bottleneck.

> Note: The source material is a forum post analyzing the arXiv survey "The Last AI Built by Humans" (arXiv 2609.11873, submitted 2025-09-10, 75 pages, CC BY-NC-ND, ~158 references, 318,000 characters of full text). One caveat stated up front: the "L7 Meta Improvement"-style seven-level labels seen in a screenshot of a Bilibili explainer video are not part of the paper — the paper uses five levels; all claims below follow the original text.

1. Identity: Who Wrote "The Last AI Built by Humans"

Corresponding author Xuanhe Zhou (Shanghai Jiao Tong University) — from systems evaluation: related works include Argus (agentic inference runtime), OmniOpt, SetupX, Workspace-Bench. Four lead authors (Yi Duan / Ying Liu / Zirui Tang / Haodong Chen) plus 30+ authors spanning SJTU, Theseus Labs, Tsinghua (Zhiyuan Liu and Bowen Zhou listed), ByteDance, Modelbest, Shanghai AI Lab, and Agent-Native Research Lab.
Bowen Zhou appears again: the author's earlier covered OpenRSI (Frontis) is a Bowen Zhou project, and he is also on this survey's author list — Appendix B's industry catalog even includes Frontis OpenRSI at L2–L3. Open-source engineering and academic survey reference each other within the same network.
The title is a manifesto: "The Last AI Built by Humans" — after which design authority is handed over. The TL;DR is blunter: "Everything AI has achieved so far is but a drop in the ocean."

2. HCI: A Unified Yardstick for "Progress"

The Headroom-Closed Index (HCI): for each benchmark, set the first-year 90th-percentile model score F₀ as 0 and perfect score as 100, H = 100×(s−F₀)/(100−F₀); domain trajectories are aggregated with √n weighting. H=0 means back at the entry frontier, H=100 means full closure. This converts incomparable benchmark scores into "headroom closure."

Readings (2026): advanced math 86.4, graduate-level science 85.8, broad knowledge 77.2, law 64.5, multimodal 62.2, frontier academic breadth 60.4 — while software engineering sits at 52.6, search & terminal agents 56.8, tool agents 39.9 (vs. cybersecurity agents at 91.9). Trajectory shapes diverge: broad knowledge annual increments 32.8→26.9→17.6 (decelerating), law 48.2→11.4→4.9 (stalling), math 32.8→53.6 (accelerating), multimodal 59.7→2.5 (saturated). Remaining headroom concentrates in "long-horizon, stateful" interactive tasks — precisely the motivating argument for RSI: gains in verifiable environments have not transferred uniformly to stateful workflows.

3. The Five-Level Framework: Partitioned by "Which Improvement Decisions AI Controls"

The survey's unit of analysis is neither models nor data but the improvement loop. Each level asks: where does the loop close, what does the successor inherit, and which decisions remain human?

| Level | Definition | Survey examples | |---|---|---| | L1 Execution autonomy | Humans set tasks/environments/update strategies; AI executes candidate improvements | FineWeb-Edu | | L2 Strategy autonomy | Goals/boundaries/evaluation fixed externally; AI diagnoses weaknesses and decides how to improve | Self-Harness | | L3 Experience autonomy | System decides what experience the next improvement round needs | SIMA 2 | | L4 Environment adaptation | Deployed interactions revise persistent state under external acceptance/governance rules | PANDO | | L5 Recursive inheritance | Persistent revisions govern the improvement mechanism itself | A-Evolve-Training, Red Queen Gödel Machine |

Appendix B classifies industry systems with these labels — of 144 level labels, L1–L3 account for ~120, L4 only 3, L5 only 11. This is the paper's hardest empirical finding: the "execution / strategy / data acquisition" segments of improvement are industrializing, while "deployment feedback looping back" (L4) and "mechanism self-modification" (L5) are nearly empty.

4. Three Challenges: Acceptance Discipline for Recursion

1. Safety inheritance: Gödel Agent rewrites its own policy and improvement logic, yet 14% of 100 MGSM optimization runs ended below the initial policy — persistence does not guarantee sustained gains. Remedies: transfer tests, version history, rollback. 2. Autonomous attribution: DGM raised SWE-bench 20%→50%, but archive maintenance and parent-selection rules were outside its self-modification scope — "generating better candidates" ≠ "improving the way candidates are discovered." RSI claims must separate AI-controlled decisions from fixed search pipelines + human acceptance criteria. 3. Reliable verification: repeated evaluator access invites gaming — the survey cites Anthropic's automated-research experiments where agents cherry-picked random seeds and tried to reverse-engineer test labels via evaluator queries; the Red Queen Gödel Machine's countermeasure: freeze the evaluator each epoch and verify with independent ground-truth anchors.

5. Theseus's Own System: Making "Environment" a Learnable Object

Of six industry practices (Theseus/Lark/Humanlaya/Modelbest/Tencent Hunyuan/Agent-Native), Theseus's four-stage co-evolution loop is most informative. Two experiments (30 tasks / 1,280 rubrics, eight frontier model-harness configurations):

Clean workspace vs. noisy workspace: all configurations win by +21.7 to +51.6pp (DeepSeek-V4-Pro+DSH: 98.2 vs 46.6)
Bare workspace vs. restructured workspace (Collection Map + Event Log): +18.65 to +39.67pp (task-level 22–24/30 wins)

Same model, same harness — only environment structure changed — beating larger-model swaps: survey-level evidence that "environment is part of the recipe, not the backdrop."

6. Cross-Check: How Two Frameworks Relate

The survey's five levels slice by "which improvement decisions AI controls"; the author's own six-layer self-improvement stack slices by "which carrier the improvement is deposited in" (weights→experience→code→harness→knowledge→direction-sense). The two are orthogonal: HGM is direction-sense in one scheme, L5 in the other. The convergent evidence matters most: the survey defines genuine RSI as "repeated, attributable, and transferable improvements in the capacity to improve" — a survey-level restatement of OpenRSI's "set the improvement rate as the optimization objective." Honesty note: NeoHorse (submitted 09-08) missed the survey's cutoff (09-10) by two days and is absent from the 12-industry-system list.

Three independent observations:

1. L4's count of 3 is a verification-bandwidth gap, not a technical one — deployment feedback requires external acceptance rules, which are exactly the recurring bottleneck (evaluator attacks, attribution difficulty); the bottleneck self-reinforces at the end of the pipeline. 2. Autonomy levels are an audit framework, not a capability ladder — the right reading of L4/L5 scarcity is "no one dares let deployment feedback directly modify persistent state." Palantir Action's commercial practice (permissions/audit/revocability first) arguably leads what the survey names. 3. The survey also cites *Externalization in LLM Agents: Memory, Skills, Protocols and Harness Engineering* (2604.08224) — "externalization / harness engineering" is becoming its own survey subject.

Honest boundaries: the survey admits the five-level boundaries have fuzzy bands (most systems span levels); all six industry practices are partner self-reports without third-party evaluation; Theseus experiments are only 30 tasks in their own workspace setup; the "Last AI" title is a narrative device — the paper's own data (L4=3, L5=11) shows genuine human exit remains far off; no code released.

Falsifiable predictions (12 months): ① first public controlled study of L4 (deployment feedback vs. offline loop on same tasks); ② "RSI-level self-reporting" appears in system READMEs plus audit tooling to check such claims; ③ the five-level framework gets marketing-distorted into an "autopilot-style LL1–L5" narrative with heavy L2 over-labeling.

---

*Verification note: arXiv HTML full text (2609.11873v1, 318K characters) directly fetched; HCI formula and all figures, five-level definitions and examples, Appendix B level-label counts, three-challenge cases, Theseus experiment tables all checked against the original; cross-confirmed via alphaXiv/DAIR.AI plus two news reprints. Project page theseus-labs-rsi.github.io is a JS shell (does not affect verification).*