The AI That Lost by 19 Points Wrote Itself a Rulebook
Start with a comparison. The same Claude Opus 5, playing Atari Pong: deployed directly, querying the model once per frame, zero memory, it lost by 19 points. Give it an external memory and a learning loop, and it comes back 21:0 across three games with different starting conditions, without dropping a single point.
The model didn't change. What changed was the layer in between.
This is Memento 3, posted to arXiv in early October—a collaboration between UCL, Huawei Noah's Ark Lab, and the University of Liverpool, with UCL's Jun Wang as corresponding author. The paper positions it as a new route toward recursive self-improvement (RSI): the model's weights stay frozen throughout; what gets modified is an external world model. Videos call it "black tech," but the actual thing is far more plain—a rulebook written in plain language that corrects itself.
Three Components
The first is the rulebook. The agent records its hypotheses about the world in natural language: the ball bounces off the paddle, doors open only with a matching-colored badge. Unobserved areas are left blank—no fabricating. It is persistent semantic memory, continuously revised as evidence arrives.
The second is the compiler. The rulebook is compiled in real time into Python code—an executable world model. During planning, the agent doesn't ask the LLM "what happens next"; it reasons directly in code: try this action sequence, predict the outcome, pick the one that clears the level. The paper gives this approach a name: code as model.
The third is the gatekeeper. Every revised piece of code must pass two checks before going live: an LLM judges whether the code faithfully implements the rulebook; then comes cell-by-cell replay—using the new code to replay historical interaction trajectories from the start, where every cell's prediction must match the actual observation exactly. One mismatch and it's rejected, revised, and re-verified. The rulebook itself is versioned git-style: the agent can dig up old hypotheses, diff two revisions, even restore versions broken by later edits.
What happens on error? Self-critique, revise the rules, recompile, pass the gate again. The loop runs until everything matches history. Nothing in this pipeline is mysterious: prediction errors are signals, rules are hypotheses, replay is the judge.
The Scorecard and the Missing Context
On the 25 public games of ARC-AGI-3, this agent cleared every one, hitting the RHAE efficiency score ceiling of 100.0 with 7,518 total actions—44% of the human baseline.
The "grand slam" is real, but one piece of background is worth adding: baseline1 on the ARC-AGI-3 community leaderboard (Rodionov, this past May, also on the executable-world-model route) also cleared all 25 games, with an efficiency score of 99.0. The gap between the two is one efficiency point, not "whether anyone has done it"—the executable world model route is already the shared foundation of the leaderboard's top two entries. Memento 3's experimental harness was itself built on baseline1's codebase. Further down the leaderboard: NOOA at 85.1 with 19 games cleared, OPINE-World at 78.4 with 20 games, and Claude Opus 5 on the standard harness at only about 40.7.
The Pong Ledger
The Pong comparison table deserves to be read in full. Model-free RL (Rainbow, IMPALA, GDI, LASER) reached 21 points, all at a learning cost of 2.0×10⁸ frames. Model-based RL (two generations of EfficientZero) hit 20–21 points at 4.0×10⁵ frames. Humans average 14.6 points. Memento 3's controller scored 21 at a learning cost of 9.5×10³ frames.
Nine thousand five hundred frames. That's roughly a 20,000-fold gap over model-free RL, and forty-fold over model-based RL—"several orders of magnitude" is a claim you can check against the table, and it's the conservative version. Learning stops at 21:0, the controller is then frozen, and the execution phase involves zero model calls. The same LLM without a world model is printed on the same table: −19.
Why It Matters
Frozen weights plus self-modifying external memory: this combination moves "self-improvement" from the parameter layer to the cognitive exoskeleton layer. The paper's own framing is a model-based RSI route: what gets improved is not the model, but the world model behind it.
This route's Achilles' heel and its highlight are the same thing: the verification gate. What the rules become is decided by the model; whether the changes count is decided by replay—if you can't reproduce history cell by cell, the change never happened. The executable world model is reliable not because it's written as code, but because it can be caught out by historical trajectories at any moment. Before those three 21:0 games came countless drafts that failed the gate.
7,518 actions, 9,500 frames, two checks. What makes this rulebook valuable isn't the rules inside it—it's the rules governing how the rules get changed.