Modifiable but Unverifiable: Three Questions on Closed-Loop RSI (ByteDance Seed × TokenWave)
> *"Most recursive self-improvement (RSI) systems discussed today are half-loop. They assume a Golden Verifier that keeps the model moving in the right direction."* — Self-Developing Agents project page
Key points
- Most RSI systems are half-loops: the improvement cycle closes, but the direction is guaranteed by an externally provided Golden Verifier. The hardest step—operationalizing a vague goal like "improve math reasoning" into a fixed task objective like "raise AIME scores"—has always been done by humans.
- The work decomposes closed-loop RSI into three questions and builds one benchmark per question: what to change (Aspire), how to learn from experience (S³Gym), how to retain changes (HarnessDev).
- Unifying evaluation philosophy: signals visible during development must be separated from signals that determine final scores. Any grade an agent can give itself does not count.
- The authors' claim is measured: the problem is not that agents cannot modify themselves, but that the tests agents apply to themselves are too weak to distinguish real gains from local gains.
- Survival funnel over 30 configuration-goal cells (5 agent configurations × 6 capability targets): 28 produced evaluable checkpoints → 21 reached qualified checkpoints → 2 exceeded the base model → 1 passed the retention threshold.
- Most striking result: Qwen3.5-4B self-training checkpoints rose steadily (0.79 → 1.75 → 2.38) while the un-evolved baseline sat at 17.86 — the best checkpoint trailed by −15.48. The authors' caption: *local gain, global regression*.
- With vague goals (48 matched Opus 4.8 runs), agents shifted time away from active training/eval (−76 min) toward deliberation and idle GPU time (+35/+37 min); trajectory density showed 2.98× more time reading task definitions, 2.39× reading eval scripts, 3.09× more LoRA/PEFT activity (LoRA usage jumped from 24.1% to 89.8%). This indicates *more goal operationalization*, not better target selection.
- Improvement is not monotonic: continued search and training can erase earlier gains, implying engineering practice must checkpoint intermediate versions and run regression comparisons.
- The single positive case (9B science goal, 45.330 → 48.000, Δ +2.670) was itself fragile: only 26/75 questions were jointly answered correctly across the two runs, with 20/75 flips.
- Treat any agent-self-assigned score as inadmissible evidence of improvement.
- Separate development-time signals from final scoring signals (dev set vs test set discipline); the authors honestly note this guarantees *separation of access and timing*, not strictly disjoint samples.
- Preserve intermediate checkpoints and run regression comparisons — self-evolution curves are not guaranteed to be monotonic.
- Run lots of attempts and expect most to roll back: 30 attempts yielded 1 retained gain.
- The framing's strength is its falsifiability: each question gets a decidable test in a ground-truth-known environment.
- Limitations flagged in the article: the Aspire behavior comparison is descriptive rather than causal; results are concentrated on specific models (Qwen3.5-4B/9B, Opus 4.8); and the single surviving positive case showed substantial question-level instability.
- The constructive takeaway for practitioners: RSI today is "modifiable but unverifiable" — the bottleneck is verification, not modification.
- Aspire: arXiv 2608.31111
- S³Gym: arXiv 2608.31100
- HarnessDev: arXiv 2609.01437
The three benchmarks
| Aspect | Benchmark | Question | Verification | arXiv | |---|---|---|---|---| | Goals | Aspire | Can agents operationalize vague goals? | Hidden 520-question expert eval with retention threshold | 2608.31111 | | Experience | S³Gym | Can self-testing/judging become self-improvement? | Seven games with executable verifiers (Chess, Minesweeper, Nullify, Tetris, Snake, PvZ, Trust) | 2608.31100 | | System | HarnessDev | Can LLMs create and evolve their own harness? | Frozen artifacts tested on held-out tasks and re-run on a fixed executor | 2609.01437 |
Aspire: what to change — only 1 of 30 cells survives
Typical failure modes documented in Aspire
1. Wrong data for the stated goal: in the 4B Self configuration, 30 of 32 data imports used GSM8K or Hendrycks math even when the goal was science, logic, or writing (11/15 on 9B). 2. Training an instruct model to collapse: 5 checkpoints ran SFT on digit-labeled MMLU — 21,000 targets were all single digits, so 279 eval outputs became single digits; HealthBench fell from 31.68 to 12.62. 3. Stopping at self-imposed criteria: Luna passed its own 8-item checklist (which had restructured mechanism/formula/parameter questions into "research design" items) and stopped searching after ~40 minutes. 4. One edit causing most of a collapse: Terra removed a hallucinating reviewer without adding a final-answer invariant; two tasks collapsed from 75.00/52.33 to 2.57/2.91, accounting for ~70% of the run's total drop. 5. Checks that never fire: Sol's final edit only checked that the last response was non-empty — it never triggered on any of 20 inspected items.
The common thread: none of these are capability failures — all are judgment failures. The agents knew how to train, edit, and write checks; what they lacked was knowing whether any of it worked.
Evidence boundary (authors' own caveat)
The vague-vs-specific comparison is a descriptive comparison of 48 matched Opus 4.8 run pairs, not an isolated prompt-only causal effect; trajectory density is normalized per 10k characters and shows where attention moved, not what caused score differences.
S³Gym: how to learn — no universal memory
S³Gym splits the experience pipeline into three stages — Self-Testing, Self-Judging, Self-Improvement — across seven text games with executable verifiers. Three candidate experience formats are compared: raw history context (History ICL), compressed summary memory (Summary Memory), and direct parameter training (Parameter Training). The core question: can agents convert self-generated experience into genuinely better next decisions, verified on decisions that have not yet been made (i.e., not certified by the trajectory that produced the experience)?
HarnessDev: how to retain — runnable is not the same as alive
HarnessDev tests whether LLMs can build a working agent harness and evolve it from downstream execution feedback. Verification requires freezing the artifact, measuring held-out generalization, and re-running under a fixed executor the agent does not control. The article links this to Aspire's "dead mechanism" failures — a check that never triggers is runnable code but not a living safety mechanism. Notably, the best evolved harness (27.22) came close to the hand-engineered Qwen-Agent reference (28.64, gap of only 1.42), though other lines trailed further (Terra 20.76, Luna 19.32) and one configuration produced no valid harness at all.