Summary
This paper introduces SPOKE (Strategic Preference Optimization with KL-anchored Equilibrium), a method designed to teach compact AI agents strategic behavior in poker under hidden information. The authors first evaluate a static baseline and three frozen language models (SmallThinker, Qwen3, Gemma-3), finding substantial performance variation between exact-game and six-player evaluations. SPOKE combines teacher-relative value penalties, policy anchoring, risk constraints, and structured rationale supervision, and is benchmarked against SFT, DPO, GRPO, and GSPO. Across three runs of a SmallThinker-4B-A0.6B policy, SPOKE achieves the lowest mean exploitability in Kuhn poker (0.213) and Leduc poker (1.081), the highest six-player top-1 consistency (0.775), while maintaining teacher KL divergence comparable to SFT (0.522 vs 0.523). The results provide preliminary evidence that strategic preference optimization can improve policy quality while preserving teacher-consistent behavior.
A poker agent can make a legal move and still make a strategically poor one. This paper asks whether compact AI agents can exhibit strategic behavior under hidden information and whether preference optimization can improve beyond static and zero-shot policies.
Method
The authors first evaluate a static baseline and three frozen language models — SmallThinker, Qwen3, and Gemma-3 — whose performance varies substantially between exact-game and six-player evaluation.
They then propose SPOKE (Strategic Preference Optimization with KL-anchored Equilibrium), which combines:
- Teacher-relative value penalties
- Policy anchoring
- Risk constraints
- Structured rationale supervision
SPOKE is compared against SFT, DPO, GRPO, and GSPO.
Results
Across three runs of a SmallThinker-4B-A0.6B policy:
- Lowest mean exploitability in Kuhn poker: 0.213
- Lowest mean exploitability in Leduc poker: 1.081
- Highest six-player top-1 consistency: 0.775
- Teacher KL divergence comparable to SFT: 0.522 vs 0.523
Conclusion
These results provide preliminary evidence that strategic preference optimization can improve policy quality while maintaining teacher-consistent behavior.
- Paper: <https://arxiv.org/abs/2610.27028>
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636295