AutoHarness: Small LLM Agents Self-Write Code Harnesses to Eliminate Illegal Moves

Forum topic · ✨步子哥 · 2026-03-21

Summary

A Chinese forum post discusses AutoHarness (arXiv:2603.03329), a method that lets lightweight LLM agents automatically synthesize their own Python 'code harness' to enforce environment rules. Large language model agents often fail in interactive settings not because of weak reasoning but because they emit illegal actions; the post cites a Kaggle GameArena chess event where 78% of Gemini-2.5-Flash's losses stemmed from illegal moves. AutoHarness uses an iterative refinement loop: the model proposes a validation program, runs it against the real environment, reads error feedback, and improves the code using Thompson sampling—converging in about 14.5 iterations on average. With its self-built harness, Gemini-2.5-Flash achieved a zero illegal-action rate across all 145 TextArena game variants, reached a 56.3% win rate in two-player games (vs. 38.2% for unshielded Gemini-2.5-Pro), and improved average single-player reward from 0.707 to 0.745. Pushing further, the model distilled entire policies into pure Python code, scoring 0.870 average reward on 16 single-player games with zero inference latency and cost, outperforming Gemini-2.5-Pro and GPT-5.2-High. The takeaway: small models with code synthesis and feedback loops can surpass much larger models.

This forum post introduces AutoHarness (arXiv:2603.03329), a technique that lets small LLM agents write their own protective code to eliminate illegal actions in game environments.

The Problem: Illegal Actions

LLM agents—language models that act step-by-step based on environment feedback—frequently fail because they violate environment rules rather than because they reason poorly. In the Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash's losses came from outright illegal moves (e.g., putting the king in check), not weak play. In TextArena, a benchmark spanning 145 single- and two-player text games, any game can trigger rule violations. Prior fixes had drawbacks:

Hand-written harnesses: effective but costly to maintain; each new environment or rule change requires new code.
Fine-tuning huge models: expensive and still imperfect.

The Method: Iterative Self-Refinement

AutoHarness has Gemini-2.5-Flash write Python code that acts as its own shield (e.g., checking whether a chess move is legal):

1. The model proposes a candidate harness program. 2. The code runs against the real environment, which returns error logs for failures. 3. The model reads the feedback and revises the code, using Thompson sampling to select promising modifications. 4. Iterations repeat until the harness is perfect—an average of only 14.5 rounds.

Results

Zero illegal-action rate across all 145 TextArena games (single- and two-player variants) after harness deployment.
In two-player games, shielded Gemini-2.5-Flash achieved a 56.3% win rate, beating the unshielded Gemini-2.5-Pro at 38.2%.
Single-player average reward rose from 0.707 to 0.745.
The harness offloads rule-following to code, freeing the model to focus on strategy—making the smaller model faster and cheaper than its larger sibling.

Pure-Code Policies

The authors pushed further by having the model distill entire game strategies into pure Python scripts—no LLM calls at decision time. This yields zero inference latency and cost, and achieved an average reward of 0.870 on 16 TextArena single-player games, outperforming Gemini-2.5-Pro and GPT-5.2-High.

Implications

The author suggests applications beyond games: self-driving validation, coding assistants avoiding syntax/security errors, and robotics. The core insight is that lightweight models, through code synthesis plus feedback loops, can self-improve enough to exceed far larger models. The post closes by inviting readers to imagine applying AutoHarness to custom game mods.

References

1. Lou, X. et al. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arXiv:2603.03329 (2026). 2. TextArena benchmark environment research (per the paper's extended description). 3. Gemini model series technical reports (Gemini-2.5-Flash vs. Pro performance, Kaggle competition data). 4. Research on code synthesis in AI agents (iterative feedback, Thompson sampling). 5. parikhakshat/AutoHarness GitHub repository: an earlier fuzzing-harness generation tool.