CodeHack Deep Dive: Rewards for Climbing the Abstraction Ladder, Insurance for Climbing Down

NetHack is a 1987 terminal dungeon crawler where a single wrong step can ruin a run lasting hundreds of thousands of turns, making it a long-horizon decision-making benchmark for AI. The naive way to have LLMs play it is to output a low-level keystroke at every step — one call to move north, one call to swing a sword. A single episode means hundreds of thousands of model calls, astronomical inference bills, and models that frequently get lost between the general store and the staircase, dying on floor one.

A paper from a Princeton and University of Warsaw team (with Benjamin Eysenbach and Karthik Narasimhan among the authors) proposes a different approach called CodeHack. The core move: recurring decisions are taken out of the model's hands and written as Python functions. explore handles map exploration, fight_melee handles close combat, pickup_armor picks up equipment, open_doors opens doors, and emergency_escape saves your life — 78 such skills, each real code wired to the keystroke-level interface, running on CPU. Instead of emitting one keystroke per step, the model picks a skill name, and the code takes over the next dozens to hundreds of steps.

Main results

Evaluated across 14 models (GPT-5, GPT-OSS-120B, the Llama family, the Gemma family, the Qwen family, uniformly over 200 episodes), with three hard metrics comparing the skill set against pure keystrokes:

• Game progress: 2.9x (1.98% vs 0.69%)
• Score: 3.8x (318 vs 84)
• Dungeon depth: 2.4x (2.88 vs 1.19 floors)
• Inference cost: from $4.35 down to $0.59 per episode — an 86% saving
• Tokens: from 6.72M down to 1.74M — a 74% saving

The mechanism is simple: model calls drop by 5.1x, and most environment steps are executed cheaply on CPU by code. This is a division of labor between judgment and execution — the model decides *what to do next*, the code decides *exactly how*. Targeted tasks show even bigger gaps: a hardcore dual-door map jumps from 28% to 97%, and on one expert-level task where the keystroke-only group went 0-for-4 across models, the skill group scored 25%.

Reinforcement learning

The RL results are even more interesting. Llama-3.1-8B and Qwen-3.5-4B were trained with PPO under the same budget across three action interfaces: the keystroke-only group gained 0.375 dungeon floors, the skill group 2.715, and the mixed group 3.24 — 7.2x and 8.6x larger gains under identical training budgets. The mechanism is classical: the action space collapses from hundreds of thousands of keystroke sequences into combinations of 78 skills, the exploration space collapses with it, and RL sample efficiency takes off. This is exactly what the options framework argued in 1999; the paper's contribution is re-quantifying it in the LLM era.

Two calibration notes:

1. 7.2x is a gain ratio under the same budget, not a wall-clock "learns 7.2x faster" — though the wall-clock learning curve also holds. 2. The 8.6x belongs to the mixed group, and part of why its gain is larger is a lower zero-shot starting point (1.81 floors vs 2.48), i.e., more room to improve. At the end of training, the skill group has the best absolute score.

The real point: leaky abstractions

The paper's real thesis is in its title: *Up and Down the Abstraction Ladder*. No skill library covers every corner — the abstract uses the word leaky. In one vault room a guard demands gold, and none of the 78 skills is a "drop gold" function; a ranged-combat skill returned without doing anything. The pure skill group gets stuck in these traps. Hence a third experimental condition: mixed control, which puts the raw keystrokes and the 78 skills into a single action space, letting the model decide at each step which rung of the ladder to work on.

In the vault, GPT-5 tried several skills to no effect, switched to primitives — dropped the gold, walked down the corridor out the door — then switched back to explore to continue mapping. The same pattern with the thrown javelin: primitives resolve the local crisis, then immediately return to the skill layer. The result: mixed control retains 95% of the skill group's progress, 91% of its score, and 97% of its depth, at the cost of 2.3x inference. The escape hatch isn't free — but without it, the gaps in abstraction become the ceiling.

Fact-check notes

The five headline numbers — 2.9x progress, 3.8x score, 2.4x depth, 86% cost saving, 74% token saving, 78 skills — all check out exactly against Table 1 and the text; no inflation. "Learning speed up 7.2–8.6x" is slightly loose phrasing (a gain ratio described as speed, two settings described as a range), and "the agent automatically downgrades" is a bit anthropomorphic — the mechanism is not an automatic trigger; it's the model *learning* to choose primitives when needed. That's precisely why the mixed group gains the most after RL: the model learned *when to climb down the ladder* too.

An overlooked, counterintuitive finding

The benefit of skills grows with model scale. In the Llama family, a 3B model digs only 1.7x deeper with the skill interface, while a 70B model digs 3.0x deeper. Scaffolding is not a crutch for weak models — it's a lever for strong ones. The stronger the model, the better it uses high-level handles.

Lineage and future work

This paper fits a lineage of "cheap at runtime" research: collapsing judgment into closed-set classification, locking reflexes into candidate sets, collapsing planning into program synthesis. CodeHack is the action-domain sample of collapsing action execution into code skills, and it proves one extra thing: narrowing the interface doesn't just save inference money — it directly amplifies learning signal. The money-saving interface and the fast-learning interface are the same interface. The paper lists future work: agents that discover recurring failures, write new skills themselves, and build their own memory-query tools — the ladder shouldn't just be climbable up and down, but extendable. That reads like a hand-written carrier of experience waiting for an RSI loop to take over.

Honest caveats

The paper states plainly: these results are far from solving NetHack, the skill library is hand-written and necessarily incomplete, and failure modes include fine-grained tactics, rare knowledge, and long-term resource management — things that don't encode well as skills. Code is MIT-licensed at github.com/BartekCupial/codehack, released only days ago.

Finally, the title's origin: the authors drew inspiration from Bret Victor's 2011 essay *Ladder of Abstraction*, which used sliders and dials to let readers slide between abstraction levels of the same phenomenon. Sixteen years later, a group of RL researchers built the same thing for language models — only this time the slider is called mixed control, and the one that learned to use it is the model itself.