Continual Harness: An AI That Writes Its Own Cheats While Playing Pokémon
> Paper: Continual Harness: Online Adaptation for Self-Improving Foundation Agents > Authors: Seth Karten, Joel Zhang, Tersoo Upaa Jr, Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, Kiran Vodrahalli > Institutions: Princeton University, ARISE Foundation, Google DeepMind > arXiv: 2605.09998v1
The Core Problem: Why Doesn't AI Get Stronger on Its Own?
Claude Code can write code and OpenHands can fix bugs, but ask a large model to play Pokémon and it can't even beat the first gym. This isn't a lack of intelligence—it's a lack of proper scaffolding.
Coding agents have mature harnesses (toolchains, memory systems, planning frameworks), but embodied agents facing long-horizon, partially observable decision-making environments have no equivalent infrastructure. Human developers must build it manually—a single harness can run to thousands of prompt lines and a dozen sub-agents.
The central question of Continual Harness: can the AI build this scaffolding itself?
Three Stages of Evolution: From Human-in-the-Loop to Fully Autonomous
Stage 1: Gemini Plays Pokémon (GPP)—Humans as the Cheat Engine
This was the team's earlier work. Human developers watched the AI play via livestreams and manually revised prompts, added sub-agents, and wrote skill scripts. After multiple iterations, GPP became the first AI system to beat Pokémon Blue, Yellow (hard mode), and Crystal.
At the hardest sections, the model began iterating on strategies through long-context memory—improving before humans made changes. This emergent self-improvement signal inspired Continual Harness.
Stage 2: Continual Harness—Removing the Human from the Loop
An automated Refiner replaces the human. Every F steps, it reads recent trajectory data, looks for failure patterns, and performs CRUD-style edits on four harness components:
| Component | Role | How the Refiner Improves It | |------|------|-----------------| | System prompt p | Strategic guidance, high-level instructions | Rewrites strategic direction | | Sub-agents 𝒢 | Specialized modules (battles, puzzles, reflection) | Adds/removes/modifies sub-agents | | Skill library 𝒦 | Reusable text heuristics and executable programs | Creates/fixes pathfinders, tool wrappers | | Memory ℳ | Facts, strategies, observations accumulated across turns | Adds/removes memory entries |
Key innovation: no environment resets. Traditional prompt optimization methods (like GEPA) must run a full episode and reset before updating. Continual Harness updates the harness in real time within a single episode—the AI learns while it plays.
Stage 3: Model + Harness Co-Learning—Closed-Loop Training
The refined harness is transferred to open-weight models. The training loop:
1. The open-weight model runs 256 steps in the live harness 2. A process reward model (PRM) scores each transition 3. Low-scoring windows are relabeled by a frontier teacher model (Gemini 3.1 Pro) 4. Soft SFT updates the model weights 5. The next iteration resumes from the previous game save—reset-free
Model weights and harness state evolve together: the harness shapes the trajectory distribution, while the model's gameplay exposes new failure modes for the next refinement round.
Experimental Results: From Blank Slate to Near-Expert
Pokémon Red and Emerald
Starting from a minimal harness (only an environment interface and a generic prompt—no sub-agents, no skills, no memory, no human knowledge), Continual Harness:
- Significantly reduces button-press cost
- Recovers most of the gap to the human-expert harness
- Correlates strongly with model capability: Gemini 3.1 Pro strictly Pareto-dominates, Flash shows high variance, and Flash-Lite falls below the capability threshold (it cannot even start the improvement loop)
- Initial: nearly 50% penalty (paying half again the optimal cost per step)
- During the run: rapidly drops to single digits and holds
- Hundreds of navigation skill calls accumulated within 24 hours of runtime
- Paper: https://arxiv.org/abs/2605.09998
- Project page: https://sethkarten.ai/continual-harness
- Prior work: Gemini Plays Pokémon (GPP)
Open-Weight Model Co-Learning
Gemma-4 (31B), trained via online co-learning inside the refined harness, achieves sustained in-game milestone progress on Pokémon Red—from both early-game and mid-game saves. The untrained baseline makes zero advance.
Quantifiable Evidence of Skill Self-Improvement
Path cost of the pathfinding skill relative to the Dijkstra optimum:
This is structured, measurable self-improvement—not vague "the model got smarter."
Key Insights
1. The Capability Threshold Is Real
Flash-Lite achieves under 20% improvement rate on Emerald, and all Continual Harness variants fail to beat the minimal baseline. Without sufficient model capability, the self-improvement loop cannot start. This is a capability problem, not an algorithm problem.
2. Being Reset-Free Matters More Than Expected
Reset-free isn't just time savings. Each reset discards environment state and game progress, forcing the AI to "re-orient." Continual Harness lets learning happen in continuous real experience rather than fragmented restarts.
3. All Four Components Must Evolve Together
Optimizing prompts alone (as with GEPA) is far from enough. System prompt, sub-agents, skill library, and memory must all be diagnosed and repaired—this is full harness state editing, not prompt tuning.
4. Open Models Can Participate, but Need a Teacher
Gemma-4 cannot yet play both the agent and teacher roles. Co-learning requires a frontier model as relabeling teacher. But as open-weight models improve, a fully self-sufficient closed loop will eventually arrive.
Significance: From "Building Scaffolding" to "AI Building Its Own Scaffolding"
The ultimate bottleneck for AI autonomy has never been the model itself, but the environment the model runs in. Every tool, prompt, and workflow humans give AI is scaffolding—and its maintenance cost keeps climbing; complex-system harnesses can reach tens of thousands of prompt tokens and dozens of parallel sub-agents.
Continual Harness points to a way out: start from a minimal interface and let the AI diagnose its own defects, write its own patches, and accumulate its own skills during operation. This applies beyond games—any long-horizon, partially observable autonomous task with complex toolchains (robot manipulation, scientific experiments, software maintenance) can benefit.
The deeper question: if AI can improve its own operating environment, could its improvement speed exceed humanity's ability to understand and control it? The paper doesn't answer this, but it pushes the question to the next crossroads.