Dream-RSI: Recursive Self-Improvement through Evolving Worlds — Deep Dive

arXiv 2609.14858 · submitted 2026-09-14 (v1) · cs.CL · 12 pages main + 24 pages appendix · 17 authors Institutions: Google / Google DeepMind / University of Maryland / University of Virginia GitHub: zhengkid/Dream-RSI (313★, code not yet released) · Project page: dream-rsi.com

0. TL;DR

One sentence: Google turns "discovery history" from static context into a replayable offline evaluation environment (a "replay simulator"), so improving the exploration strategy changes from expensive online trial-and-error into zero-execution-cost scoring over historical replay — essentially porting the model-based RL Dreamer idea to the meta-level of LLM discovery systems. Engineering-wise clever and experimentally effective, but since the "worlds" only cover realized history, policy improvement is structurally confined to re-navigating the old map.

Three core judgments:

1. The problem is real and the solution targets it. "Meta-level feedback is delayed and expensive" is a genuine bottleneck for long-horizon discovery systems (evaluating an exploration strategy requires observing hundreds of proposal-evaluation cycles). Using recorded discovery trees for deterministic replay to evaluate candidate policies offline — one expensive online run supporting thousands of free off-policy evaluations — is the paper's most solid lever. 2. The controlled experiment is clean, but head and tail have caveats. Compared against a Recursive Fixed Exploration baseline sharing everything (same agent/evaluator/initialization/budget, differing only in whether the policy improves across rounds), gains are credible: Lasso saves 1.7× calls with better quality; kernel tasks save 1.79–2.43× generations or achieve 1.44–2.09× higher performance at equal budget. But the "162×/two orders of magnitude" comparison vs. SimpleTES is confounded by systematic differences. On the math tasks, Dream-RSI scores 1.456375 on Autocorrelation — strictly worse than its own fixed-policy baseline (1.456001, lower is better); Sum-Diff gains are in the fourth decimal place. The paper glosses this as "competitive." 3. RSI positioning: another instance of bounded L5. By the Theseus survey's taxonomy (2609.11873), Dream-RSI improves exactly one of L5's four objects (search policy), but the improver itself (LLM policy-development agent + replay argmax selection) is fixed, and the "monotonic improvement guarantee" holds only on replay scores over fixed history — same category as HGM: bounded L5. The paper and the survey were submitted 3 days apart; neither knew of the other.

1. The Problem

LLM-driven discovery systems (the AlphaEvolve/FunSearch family) run iterative discover loops: generate candidates, evaluate, absorb feedback, regenerate. As targets get harder, discovery requires long-horizon exploration spanning thousands of proposal-evaluation cycles. What determines whether compute is well spent is the orchestration of exploration: how many branches to open, how deep to refine each, how to batch parallel work, whether to rescue failed branches, when to stop.

The dilemma (paper §1):

• Fixed policies (the default in the Evolve family) cannot learn from accumulated experience, repeatedly wasting compute on ineffective directions;
• Online policy optimization (e.g., EvoX) requires evaluating long-horizon consequences of a strategy — feedback is delayed and expensive, and the meta-policy space is huge, so trial-and-error itself burns online budget.

The paper's intuition: if you had a fast, cheap "simulator of the discovery process," you could evaluate hundreds of candidate policies before expensive online deployment. And that simulator is already lying in your hands for free — the completed history itself.

2. Core Idea: Discovery History = Replay Simulator ("Worlds")

2.1 The analogy

A navigation analogy (§2): the first pass through unfamiliar terrain takes wrong turns and dead ends, but the paths walked become a map; any new navigation policy can now plan on the map without revisiting the field. This corresponds to model-based RL / world models (Sutton 1990; Ha & Schmidhuber 2018; the Dreamer family, Hafner 2019–2025): learn an environment dynamics model and "dream" in it to train policies.

Dream-RSI's dreaming happens not in a learned parametric model but in a real, recorded, non-parametric tree — no model error, but limited support (see §6.1).

2.2 Discovery tree: reducing free exploration to a "branch × attempt grid"

The discovery tree is rooted in an initial workspace; each non-root node records the full result of one "generate-evaluate" attempt: filesystem snapshot, generated artifacts, evaluation diagnostics, score s_v.

A key shared decision interface (used identically online and offline): the policy observes the currently revealed tree and selects a batch C (≤W parallel workers) from the selectable node set A(T) = {root} ∪ {leaves}; the batch simultaneously decides "where to continue" and "how many to parallelize."

A frequently missed structural constraint: online, each node has exactly one recorded child continuing it (linear workspace refinement), so the entire history is reduced to what the prompt calls a "frozen, irregular branch×attempt grid." The policy's real degrees of freedom shrink to four things: which roots to open, how deep each goes, how batches are composed, when to stop. Engineering "infinite policy space" into a finite, replayable, comparable scheduling space is clever — but it caps the ceiling of "improvement" (see §6.2).

2.3 The three-stage loop (Figure 1)

1. ❶ Online Explore: the current policy (executable Python code) drives a fixed discovery agent (Gemini CLI running Gemini-3.1-Pro / 3.7-Flash) to expand the discovery tree; a fixed evaluator scores. Rollouts run at most K₁ decision rounds; the completed tree T_t is appended to history H_t. 2. ❷ Construct Replay Simulator: each historical tree becomes a replay "world" in a pool. 3. ❸ Dreaming-based Policy Improvement: offline, M policy versions are constructed and evaluated. Each is replayed on all historical trees: starting from a root-only revealed subtree, the policy selects batches → deterministically receives the recorded children (no new candidates generated) → revealed observations feed the next step. A fixed LLM policy-development agent reads replay trajectories and scores, revising the policy code to produce the next version. Finally π_{t+1} = argmax_m V^m is deployed, returning to ❶.

The only thing that changes is the exploration policy code; base models, evaluator, and execution interface are all fixed.

2.4 Replay objective

\[V_i^m = \underbrace{\max_{v} s_v}_{\text{discovery quality}} - \underbrace{\beta_1 N_i^m}_{\text{execution cost}} + \underbrace{\beta_2 \frac{N_i^m}{\max\{1,k\}}}_{\text{parallelism bonus}}\]

N is the number of revealed non-root nodes (the generate-evaluate requests implied by the trajectory), k the number of decision rounds. The third term rewards "attempts executed in parallel per round" — parallel workers don't consume extra wall-clock online, so the replay objective explicitly favors batching policies (serial policies penalized ≈1, full batches ≈1/W). A version's total score is the mean replay score over history.

2.5 The monotonic improvement guarantee — and its boundary

Since the candidate set includes the current policy, selection satisfies V^{m*} ≥ V^0: the selected policy is no worse on replay scores over fixed history. This is the paper's only "guarantee," and its boundary must be understood precisely:

• It guarantees replay-score monotonicity, not online-performance monotonicity;
• The generalization gap between replay and online performance (off-policy → on-policy) has no theoretical characterization, only empirical backing (online curves in Fig 3b / Fig 4 / Fig 6).

2.6 Anti-cheating: the prefix-only interface

The policy may use only prefix-observable information during replay: callable APIs like question.observed() (revealed prefix), legal_actions(), probe_batch(); hard constraints forbid using unrevealed scores, hard-coding winning node IDs, or absolute score targets. The policy-development agent also may not copy specific traces' branch/cell IDs/scores into policy logic (prompt-level soft constraint). This protection is prompt-level, with no formal verifier — leak prevention rests on discipline rather than mechanism (see §6.4).

3. Experiments and Evidence Strength

3.1 Tasks and setup

| Domain | Task | Model | Rounds | |---|---|---|---| | Algorithm engineering | Lasso regularization-path solver (SimpleTES benchmark, 17 training instances + 6 held-out downstream datasets) | Gemini-3.1-Pro (10 parallel × 11 refine = 110 calls/round), Gemini-3.7-Flash (32×20=640/round) | 5 | | Mathematical optimization | Sum-Difference / Autocorrelation / Circle Packing (n=26, 32) | Gemini-3.1-Pro | 10 | | GPU kernels | KernelBench: VGG16 / LayerNorm / ConvDiv / ConvMax | Gemini-3.1-Pro | recursive multi-round |

Controlled baseline Recursive Fixed Exploration: same discovery agent, evaluator, initialization, resource constraints; only the policy is unchanged across rounds. Both methods behave identically in round one — all subsequent differences attribute to "policy improvement" itself. This is the cleanest causal design seen in papers of this kind.

3.2 Results, item by item

Lasso (main experiment, highest credibility):

• Gemini-3.1-Pro: Dream-RSI uses 317 calls (baseline 550) to reduce held-out average runtime from 3587.1 ms to 2931.0 ms — 42% fewer calls while being faster;
• Gemini-3.7-Flash: 1879 vs. 3200 calls; 2516.7 → 2350.6 ms;
• Both discovered solvers beat sklearn and glmnet on all 6 held-out datasets;
• The discovered solver (full source in Appendix C, C++/Eigen) has real technical substance: strong-rule screening + adaptive Cauchy–Schwarz KKT pruning + disjoint active-set bookkeeping + lazy Gram matrix + hardware-aware optimization — a third path distinct from SimpleTES's "switch LARS/coordinate descent per problem dimension," especially strong on large-scale matrices like RCV1.

Discount the "162×" vs. SimpleTES: SimpleTES (GPT-OSS-120B, 51,200 generations) was a one-shot run; Dream-RSI is a recursive multi-round structure with different architecture, models, and evaluation protocols; the SimpleTES vs. SimpleTES† rows differ hugely (avg 3804.8 vs. 8318.4) with the † meaning unexplained in the extracted text. "Two orders of magnitude budget saving" is acceptable directionally, not as a precise multiplier.

Math tasks (weakest evidence):

• Sum-Diff (↑): 1.145427, above SimpleTES (1.143975) and RFE (1.144047) — but the margin 0.0012 sits in the fourth decimal; no variance/seed sensitivity reported;
• Circle Packing: 2.635983, tied with the best reported result (already saturated across systems);
• Autocorrelation (↓): 1.456375 loses not only to SimpleTES (1.453675) but strictly to its own fixed-policy baseline RFE (1.456001). The paper's wording: "remains competitive." This item is effectively a counterexample: the policy selected by replay-score argmax performed worse online — exactly the generalization gap of §2.5, undiscussed in the paper.

GPU kernels (credible): VGG16 / LayerNorm save 2.43× / 1.79× generations at equal performance; ConvDiv / ConvMax gain 2.09× / 1.44× performance at equal budget.

3.3 Two high-value analysis experiments (§5)

1. History-as-simulator beats history-as-semantic-guidance (Fig 5): the natural alternative — abstracting history into high-level "directional insights" injected into prompts — is consistently worse, for both fixed policies and Dream-RSI. Strong inductive biases over-constrain the search space in long-horizon discovery. Independently valuable finding: history's value lies not in "summarized lessons" but in "replayable detail." 2. Learned adaptive behavior (Fig 6, ConvDiv): across 9 rounds, round-best performance rose 0.427 → 1.898 (4.4×); the policy contracts compute during performance-rise phases (evaluation attempts 110→50), then re-expands (80–91) after plateau, in sync with renewed performance gains. The meta-level genuinely learns non-trivial compute scheduling, not noise.

3.4 Cost-accounting blind spot

"317 calls" counts only discovery-agent calls. The policy-development agent's M revisions × replay reading over t worlds, plus orchestration overhead, are entirely unreported. Replay reading should be cheap, but M×t LLM context calls are not free. The claimed "discovery cost reduction" should be read as "online discovery budget reduced, meta-level cost externalized." A common flaw in this genre (DGM, Meta-harness share it), but worth naming.

4. Positioning within the RSI Lineage

4.1 Meta-level peers cited (one line in related work)

"Recent work has begun to optimize meta-level mechanisms (Kim et al. 2026; Liu et al. 2026a; Wang et al. 2026; Yan et al. 2026a; Zhang et al. 2026c)":

| Citation | System | Difference from Dream-RSI | |---|---|---| | Kim et al. 2608.24735 | Meta^n | recursive emergent-depth self-improvement; Dream-RSI is single-level meta-improvement | | Liu et al. 2602.23413 | EvoX | also optimizes the search strategy itself, but evaluates policies online; Dream-RSI's differentiator is offline replay evaluation. No head-to-head experiment — the nearest direct competitor missing from comparison is the biggest experimental gap | | Wang et al. 2607.05297 | MetaSkill-Evolve | skill-level two-timescale evolution | | Yan et al. 2605.08039 | PACEvolve++ | long-horizon progress-aware evolution | | Zhang et al. 2603.19461 | Hyperagents | (Theseus's L5 candidate) general hyperagents |

Also cited: DGM (Zhang & Clune, ICLR 2026), Meta-harness (Lee et al. 2603.28052), SkyDiscover, SwarmResearch.

4.2 Taxonomy placement

• Improved object: search policy (exploration-orchestration code) — exactly one of Theseus L5's four objects;
• Improver: fixed LLM policy-development agent + replay argmax selection — the improvement mechanism itself is not improved;
• Inheritance: discovery-tree history (growing) + policy code version chain;
• Verdict: bounded L5, comparable to HGM. It also exposes two L5 pathologies: ① argmax inheritance is monotone on replay scores, not guaranteed online (Autocorrelation counterexample); ② cross-task generalization untested — each task's policy improves independently; no "learned exploration policy transfers to new tasks" experiment.

4.3 The lineage it should align with but doesn't: OPE

Dream-RSI is essentially bringing off-policy evaluation (OPE) to the meta-level of LLM discovery systems: evaluating alternative policies on recorded decision-outcome data. Unlike importance-sampling/doubly-robust OPE, this is deterministic replay of real records — no estimation variance, no model error, but can only evaluate policies within the historical support set. The paper cites no OPE literature; the analogy lands only on Dreamer/world models. This gap matters: OPE's support-collapse and coverage metrics (e.g., concentration coefficients) are precisely the formal language for the §6.1 critique.

5. Gold in the Implementation Details (Appendix B prompts)

1. Online exploration prompt (B.1): instructs the agent to "read the full history, not samples," "distinguish bad ideas from good ideas dragged down by bugs (the latter retryable only after locating the specific bug)," "identify local-optimum clusters and force structural divergence." A model of discovery-system prompt engineering. 2. Policy-improvement prompt (B.2) reveals the policy's true shape:

• A scalar beta knob (fixed within a round, adaptive across rounds): high beta = wider, more patient, weaker pruning; low beta = less exploration, earlier stopping, stronger pruning. A cross-round default with evidence-driven adjustment rules (rising performance → keep; plateau + sweep favors high beta → +0.1~0.2; high beta tried and ineffective → lower; insufficient evidence → 0.6);
• plan_grid(branch_count, refine_count) plans the grid shape before each online round;
• A failure taxonomy: hard-unrecoverable / repairable implementation failure / weak-but-underexplored / repeatedly hopeless — "compile error ≠ algorithmic failure; a single repairable failure must not permanently starve a branch";
• Batch-composition rules: dynamically mix exploitation + exploration + at most one recovery; fixed quotas, randomness, and "only pick the single best" are forbidden.

So "recursively improving the exploration policy" mostly reduces to learning: beta defaults, grid shapes, ranking and pruning thresholds, batch-composition logic. Limited expressiveness, but replayable, comparable, monotonically selectable — a design tradeoff, not a defect; understanding this sets correct expectations for the framework's capability boundary.

6. Critical Assessment

6.1 Support-set truncation of the "worlds" — the fundamental limitation

Replay returns the recorded child for a chosen node; branches absent from history return ∅ and the policy must stop. Dreaming can therefore only evaluate rearrangements of the realized search space: different branch choices, orders, parallel groupings, stopping points. A genuinely novel policy (opening directions that never existed in history) has its incremental value invisible in replay — precisely truncated by support limits.

This is the essential difference from Dreamer: Dreamer's parametric world model can (with error) extrapolate to unvisited states; Dream-RSI's non-parametric world has zero error but zero extrapolation. Implication: as rounds accumulate, policy improvement increasingly favors "better mining of the old map" while "opening new continents" is systematically undervalued — a structural conservatism bias. The parallelism bonus and "new-roots-first" prompts partially mitigate; mechanically unsolved. Long-term, when the history pool is large enough this matters less (online exploration keeps expanding the pool), but it frames the framework as improving exploration efficiency rather than exploration horizon.

6.2 "Evolving Worlds" oversells slightly

What evolves is the simulator pool (grows each round); each individual "world" (history tree) is frozen — and further reduced to a linear-refinement branch grid (§2.2). Strictly: "frozen grid worlds, growing pool." Demystify the marketing name when reading.

6.3 Replay objective vs. online objective mismatch

The replay objective V = max s_v − β₁N + β₂(N/k) differs from the online metrics used (held-out performance, final score, cumulative calls). Especially the max term: replay rewards "finding the best node within the historical tree," which may reward policies fitting the historical distribution rather than the unsampled online space. The Autocorrelation counterexample (§3.2) is consistent with this. β₁, β₂ values are not disclosed; replication cannot be calibrated.

6.4 Leak prevention is discipline, not mechanism

The prefix-only constraint lives in prompts, with no static checker verifying policy code doesn't use private information (e.g., implicit overfitting that "coincidentally always picks historical winners"). The policy-development agent has read replay trajectories and beta sweeps — the overfitting path is open. Mitigating evidence: continued online improvement (Fig 3b/4/6) — pure overfitting wouldn't keep winning online. But Autocorrelation shows the risk isn't zero.

6.5 Statistical discipline

All experiments are single runs, no seed repeats, no error bars. Fourth-decimal differences cannot be distinguished from noise under this discipline. This is field-wide practice in 2026 discovery-system papers — not a special knock on Dream-RSI — but readers should know.

6.6 Four things the paper didn't do

① Head-to-head with EvoX (nearest peer); ② cross-task policy transfer; ③ full accounting of meta-level costs; ④ any safety/runaway discussion (an RSI paper in 2026 has no excuse for omitting a safety section entirely; the only safety trace is "no pkill/killall" engineering hygiene in prompts).

7. Highlights Worth Stealing

1. "Use history as environment, not context" — a paradigm shift transferable to any accumulating agent system; §5.1 further shows the common "history → semantic summary → prompt" route underperforms "history → replay environment," counterintuitive and important. 2. The controlled baseline design: Recursive Fixed Exploration makes "policy improvement" the only variable; round one is identical for both. Anyone doing recursive-improvement experiments should copy this. 3. Argmax monotone selection: including the current policy in the candidate set guarantees no replay-score regression — one design line buys a "never worse" floor; extremely high value-for-effort. 4. Parallelism in the objective (β₂·N/k): explicitly encoding wall-clock efficiency into the offline evaluation target, bridging the offline-score/online-cost gap. 5. Beta knob + plan_grid: compressing the unmanageable "policy code space" into a few scannable, interpretable, cross-round schedulable knobs — a stable hybrid of LLM code-editing + grid search. 6. Failure taxonomy (repairable vs. algorithmic) written into the policy spec — turning the bandit literature's "be optimistic about repairable failures" into an executable interface.

8. Reproduction and Open-Source Status

• GitHub zhengkid/Dream-RSI: 313★ / 15 forks / 4 commits, containing only the paper PDF, project page link, citation file; full codebase, discovered programs, and reproduction scripts all marked "Being prepared"; no LICENSE;
• Appendix C gives the discovered Lasso solver's complete C++/Eigen source (the only runnable artifact right now);
• Key hyperparameters β₁/β₂/M/K₂ not disclosed in the extracted text;
• Verdict: currently not reproducible; credibility rests entirely on the paper's numbers and author credentials (Google/GDM). If code releases, the Autocorrelation counterexample and replay-online gap deserve re-examination.

9. Overall Rating

| Dimension | Rating | |---|---| | Problem choice | ★★★★★ delayed/expensive meta-level feedback is a real bottleneck, previously unexploited | | Core idea | ★★★★☆ history→replay-simulator is clean and powerful; but non-parametric replay's support truncation is a hard ceiling | | Experimental design | ★★★★☆ extremely clean controlled baseline; missing EvoX comparison, no seed repeats | | Evidence strength | ★★★☆☆ Lasso/kernels credible; one math counterexample, one fourth-decimal result | | Engineering disclosure | ★★★☆☆ full prompts are a plus; missing hyperparameters, unreleased code | | RSI significance | ★★★★☆ a clear bounded-L5 instance; bringing OPE thinking to discovery meta-levels, likely to be heavily borrowed |

Three action items for RSI lineage trackers:

1. Dream-RSI now shares a citation cluster with Meta^n/Metan, and is in direct (unacknowledged) competition with EvoX — watch who merges "replay-based meta-improvement × online meta-evolution" first; 2. Theseus's "bounded L5" judgment applies; the Autocorrelation counterexample provides a ready-made case that "replay-score monotonicity ≠ online monotonicity," usable as a yardstick for other argmax-inheritance systems; 3. The lever (history → offline evaluation environment) is discovery-system-agnostic and can be grafted onto AlphaProof-Nexus, the Drosophila project, or any long-horizon agent — hence Google built it as a lightweight orchestration layer with the base agent untouched: it aims to be a meta-layer add-on for the whole Evolve family, not another Evolve variant.

---

Quick Facts Card

• Submitted 2026-09-14 00:10 UTC, v1, 777KB; corresponding: xidongwu@ / zzhangx@ google.com; first author Tong Zheng (Google + UMD; also author of Parallel-R1 and LLMs-improving-LLMs 2605.08083; Google agentic discovery line)
• Base: discovery agent = Gemini CLI running Gemini-3.1-Pro / Gemini-3.7-Flash; policy = Python OptimalPolicy.solve() + plan_grid()
• 8 tasks: Lasso path, Sum-Diff, Autocorrelation, Circle Packing (26/32), VGG16, LayerNorm, ConvDiv, ConvMax
• Headline numbers: Lasso saves 1.7× calls (vs. RFE) / 162× (vs. SimpleTES, with architectural confounds); kernels save 1.79–2.43× generations or +1.44–2.09× performance; math tasks: two ties, one win, one loss
• Third-party coverage: alphaXiv / HuggingFace papers / Moonlight (all descriptive, no independent critical review); Turing Post's "9 Paths Toward True RSI" predates the paper and doesn't include it