Feedback That Backfires in Games: LLM Players Repeat Failed Moves After Error Records

Forum topic · 小凯 · 2026-10-24

Summary

A new arXiv paper (2610.30205) by Esmail Gumaan examines whether writing a failed move and its error message back into an LLM agent's transcript makes the model repeat that failed move in games. Testing Sokoban, Connect Four, and Othello — where failures are either illegal moves or legal-but-losing moves, both exactly verifiable — the study finds that across 8 instruction-tuned models (135M–1.7B), the record actually increases the probability of the failed move (corrective gain G = -1.38 nats/token), driven by the move's surface form. The effect holds for illegal moves but not losing moves. After the record, a move's log-probability barely depends on its prior likelihood (slope 0.18); greedy decoding repeats 38% of moves the model was about to play. Omitting the move string from the failure description preserves model preferences. Along the OLMo 2 1B post-training ladder, the inversion exists in the base model and is largely removed at the DPO stage. RAGEN default models invert before any RL. 'Do not repeat' instructions are ineffective, clean retries are the worst harness, and decoder bans stop exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU.

Paper Overview

Field: Machine Learning Author: Esmail Gumaan Published: 2026-10-24 arXiv: 2610.30205

Abstract

When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. This paper tests whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly.

Key Findings

On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M–1.7B), it makes the failed move more likely (corrective gain G = -1.38 nats per token).
The effect operates through the move's surface form; it holds for illegal moves but not for losing ones, and not for every model.
After the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them.
A description of the failure that omits the move string keeps the model's preference instead.
Along the OLMo 2 1B post-training ladder, the inversion is present in the base model and largely removed at the DPO stage.
RAGEN's default models invert in RAGEN's own Sokoban format before any RL.
A "do not repeat" instruction does nothing; a clean retry is the worst harness; a decoder ban stops exact repeats but not illegal play.
All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.