Continual Harness: AI Writes Its Own 'Cheat Codes' While Playing Pokémon
> Paper: *Continual Harness: Online Adaptation for Self-Improving Foundation Agents* > Authors: Seth Karten, Joel Zhang, Tersoo Upaa Jr, Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, Kiran Vodrahalli > Institutions: Princeton University, ARISE Foundation, Google DeepMind > arXiv: 2605.09998v1
---
The Core Problem: Why Doesn't AI Get Stronger on Its Own?
Claude Code can write code and OpenHands can fix bugs, but ask a large model to play Pokémon and it cannot even beat the first gym. The issue isn't intelligence—it's the lack of proper scaffolding.
Coding agents have mature harnesses (toolchains, memory systems, planning frameworks). Embodied agents facing long-horizon, partially observable decision environments have no equivalent infrastructure. Human developers build these manually—a single harness can span thousands of lines of prompts and a dozen sub-agents.
The central question of Continual Harness: can the AI build this scaffolding itself?
---
Three Stages of Evolution: From Human-Guided to Fully Automated
Stage 1: Gemini Plays Pokémon (GPP)—Humans as the Cheat Code
The team's earlier work: human developers watched the AI play via livestreams and manually revised prompts, added sub-agents, and wrote skill scripts. After multiple iterations, GPP became the first AI system to complete Pokémon Blue, Yellow (hard mode), and Crystal.
At the hardest sections, the model began iterating on strategies via long-context memory—before humans intervened. This emergent self-improvement signal inspired Continual Harness.
Stage 2: Continual Harness—Removing the Human from the Loop
An automated Refiner replaces the human. Every F steps, the Refiner reads recent trajectory data, identifies failure patterns, and performs CRUD edits on four harness components:
| Component | Role | How the Refiner Improves It | |------|------|-----------------| | System prompt p | Strategic guidance, high-level instructions | Rewrites strategic direction | | Sub-agents 𝒢 | Specialized modules (battle, puzzle-solving, reflection) | Adds/removes/modifies sub-agents | | Skill library 𝒦 | Reusable text heuristics and executable programs | Creates/fixes pathfinders, tool wrappers | | Memory ℳ | Accumulated facts, strategies, observations | Adds/removes memory entries |
Key innovation: no environment resets. Traditional prompt-optimization methods (e.g., GEPA) run a full episode and then reset before updating. Continual Harness updates the harness in real time *within* a single episode—the AI learns while playing and plays while learning.
Stage 3: Model + Harness Co-Learning—Closed-Loop Training
The refined harness is transferred to an open-source model. The training loop:
1. The open model runs 256 steps in the live harness 2. A process reward model (PRM) scores each transition 3. Low-scoring windows are relabeled by a frontier teacher model (Gemini 3.1 Pro) 4. Soft SFT updates the model weights 5. The next iteration resumes from the previous round's game save—reset-free
Model weights and harness state evolve together: the harness shapes the trajectory distribution, while the model's gameplay exposes new failure modes for the next refinement round.
---
Results: From Blank Slate to Near-Expert
Pokémon Red & Emerald
Starting from a minimal harness (only the environment interface + a generic prompt—no sub-agents, skills, memory, or human knowledge), Continual Harness:
- Substantially reduces button-press cost
- Recovers most of the gap to the human-expert harness
- Correlates strongly with model capability: Gemini 3.1 Pro strictly Pareto-dominates, Flash shows high variance, and Flash-Lite falls below the capability threshold (it cannot even start the improvement loop)
- Initial: nearly 50% penalty (half again as costly per step)
- During the run: quickly drops to single digits and stays there
- Hundreds of navigation skill calls accumulated within a 24-hour run
- Paper: https://arxiv.org/abs/2605.09998
- Project page: https://sethkarten.ai/continual-harness
- Prior work: Gemini Plays Pokémon (GPP)
Open-Source Model Co-Learning
Gemma-4 (31B), trained via online co-learning in the refined harness, achieves sustained in-game milestone progress on Pokémon Red from both early- and mid-game saves. The untrained baseline makes zero advance.
Quantifiable Evidence of Skill Self-Improvement
Path cost of the pathfinding skill relative to the Dijkstra optimum:
This is structured, measurable self-improvement—not a vague "the model got smarter."
---
Key Insights
1. The Capability Threshold Is Real
Flash-Lite improves by less than 20% on Emerald; every Continual Harness variant underperforms the minimal baseline. If model capability is insufficient, the self-improvement loop never starts. This is a capability problem, not an algorithm problem.
2. Being Reset-Free Matters More Than Expected
Reset-free isn't just about saving time. Every reset discards environment state and game progress, forcing the AI to "re-orient." Continual Harness lets learning happen in real continuous experience rather than fragmented restarts.
3. All Four Layers Must Evolve Together
Optimizing prompts alone (as GEPA does) is far from enough. System prompt, sub-agents, skill library, and memory must all be diagnosed and repaired—this is full harness state editing, not prompt tuning.
4. Open Models Can Participate, But Need a Teacher
Gemma-4 cannot yet play both agent and teacher. Co-learning still requires a frontier model as the relabeling teacher. But as open models improve, a self-sufficient closed loop is coming.
---
Significance: From "Building Scaffolding" to "AI Building Its Own Scaffolding"
The ultimate bottleneck for AI autonomy is not the model itself but the environment it runs in. Every tool, prompt, and workflow humans give AI is scaffolding—and its maintenance cost keeps growing; complex-system harnesses can reach tens of thousands of prompt tokens and dozens of parallel sub-agents.
Continual Harness shows a way out: starting from a minimal interface, let the AI diagnose its own defects, write its own patches, and accumulate its own skills during operation. This applies beyond games—any long-horizon, partially observable autonomous task requiring complex toolchains (robotic manipulation, scientific experiments, software maintenance) can benefit.
The deeper question: if AI can improve its own operating environment, could its improvement rate outpace humanity's ability to understand and control it? The paper doesn't answer this—but it pushes the question to the next intersection.
---