[论文] Teaching AI Agent to Play Poker Strategically Through Preference Optim...

论文概要

研究领域: NLP 作者: Phurinat Polasa, Thanapon Thanadunpremdet, Phatthanasak Kraiduang, Saksorn Ruangtanusak 发布时间: 2026-10-22 arXiv: 2610.27028

中文摘要

一个扑克智能体可以做出合法的动作，但仍然是一个策略上糟糕的动作。我们探讨紧凑的AI智能体能否在隐藏信息下展现出策略性行为，以及偏好优化能否超越静态和零样本策略。我们首先评估了一个静态基线和三个冻结的语言模型：SmallThinker、Qwen3和Gemma-3，它们在精确对局评估和六人评估之间的表现差异很大。因此我们提出SPOKE（带KL锚定均衡的策略偏好优化），结合教师相对值惩罚、策略锚定、风险约束和结构化推理监督，并与SFT、DPO、GRPO和GSPO进行比较。在SmallThinker-4B-A0.6B策略的三次运行中，SPOKE在Kuhn（0.213）和Leduc（1.081）扑克博弈中取得了最低的均值可利用度，以及最高的六人局top-1一致性（0.775），同时保持与SFT相当的教师KL散度（0.522 vs 0.523）。这些结果为策略偏好优化可以在保持教师一致行为的同时提升策略质量提供了初步证据。

原文摘要

A poker agent can make a legal move and still make a strategically poor one. We ask whether compact AI agents can exhibit strategic behavior under hidden information and whether preference optimization can improve beyond static and zero-shot policies. We first evaluate a static baseline and three frozen language models: SmallThinker, Qwen3, and Gemma-3 whose performance varies substantially between exact-game and six-player evaluation. We therefore propose Strategic Preference Optimization with KL-anchored Equilibrium (SPOKE), combining teacher-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision, and compare it with SFT, DPO, GRPO, and GSPO. Across three runs of a SmallThinker-4B-A0.6B policy, SPOKE achieves the lowest mean exploitability in K...

#论文 #arXiv #NLP #小凯