[论文] Feedback That Backfires in Games: Do LLM Players Repeat the Move They ...

论文概要

研究领域: ML 作者: Esmail Gumaan 发布时间: 2026-10-24 arXiv: 2610.30205

中文摘要

LLM 智能体一步走失败后，harness 会把走法和失败信息写进 transcript 再问一次——此前在工具调用中已发现，这种记录会让小模型更可能重复失败动作。本文检验游戏中是否同样成立（推箱子、四子棋、黑白棋），走法可因非法或合法但落败而失败，两者都有精确验证。在由正确答案扰动构建的固定探针项上，记录确实适得其反：汇总 8 个指令微调模型（135M–1.7B），失败走法概率反升（corrective gain G = −1.38 nats/token），机制是走法的表面形式；对非法走法成立、对落败走法不成立，且并非每个模型如此。探索性分析揭示了符号的来源：记录之后，失败走法的对数概率几乎不取决于它之前的概率（斜率 0.18）——模型本不会走的走法变得更容易出现，贪心解码开始产出它们；本要走的走法变得不太可能，但贪心解码仍有 38% 在重复。省略走法串、只描述失败的记录则保留模型偏好。沿 OLMo 2 1B 后训练阶梯，反转存在于基座模型、在 DPO 阶段基本消除。RAGEN 默认模型在未做 RL 时、在其自身 Sokoban 格式下就出现反转。「do not repeat」指令无效，干净重试是最差的 harness，decoder ban 能止住精确重复但止不住非法走法。全部测量在免费 Colab GPU 上以 float32 完成，首先复现了原论文的数字。

原文摘要

When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. We test whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M-1.7B) it makes the failed move more likely (corrective gain G = -1.38 nats per token), through the move's surface form, for illegal moves but not for losing ones, and not for every model. An exploratory analysis shows what sets the sign: after the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely, and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them. A description of the failure that omits the move string keeps the model's preference instead. Along the OLMo 2 1B post-training ladder the inversion is present in the base model and largely removed at the DPO stage. RAGEN's default models invert in RAGEN's own Sokoban format before any RL. A "do not repeat" instruction does nothing, a clean retry is the worst harness, and a decoder ban stops exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.

---

*自动采集于 2026-10-24*

#论文 #arXiv #ML #小凯