Quantum Frog: Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game

论文概要

研究领域: ML
作者: Saad Mankarious
发布时间: 2026-05-26
arXiv: 2505.21639

中文摘要

我们介绍了Quantum Frog，一个基于新颖量化时间机制的双人合作游戏，其中环境仅在玩家行动时前进。受经典街机游戏Frogger启发，Quantum Frog需要两只青蛙穿越8×8的交通网格并一起到达远侧。我们使用强化学习(RL)作为分析视角来回答四个设计问题：(1)游戏难度如何随交通密度缩放，(2)最优单代理策略是什么以及为什么，(3)独立和合作双人游戏之间的合作差距有多大，(4)当代理被激励合作时会涌现出什么联合策略？我们分五个升级阶段训练代理：表格Q学习、深度Q网络(DQN)、独立DQN(IDQN)和多智能体近端策略优化(MAPPO，带集中式critic)，评估每种方法对1到6辆车的交通密度。我们的关键发现是：(i)量化时间机制使冲刺策略（每步直接向上移动）普遍最优，因为它最小化了暴露在交通中的时间；(ii)添加不协调的第二名玩家比将单专家玩家的交通增加六倍更难；(iii)合作训练比独立代理恢复了32-34个百分点的联合成功率，并将回合长度从90步减少到6步；(iv)涌现的合作策略是同步冲刺，而非复杂的位置协调，说明共享激励本身足以在时间关键的合作任务中对齐代理。这些发现为Quantum Frog的商业设计提供了具体、经验验证的指导，并为环境机制在塑造多智能体学习动态中的作用提供了更广泛的见解。

原文摘要

We introduce \emph{Quantum Frog}, a two-player cooperative game built on a novel \emph{quantized-time} mechanic in which the environment advances only when a player acts. Inspired by the classic arcade game Frogger, Quantum Frog requires two frogs to cross an 8$ imes\(8 grid of traffic and reach the far side together. We use reinforcement learning (RL) as an analytical lens to answer four design questions: (1) how does game difficulty scale with traffic density, (2) what is the optimal single-agent policy and why, (3) how large is the cooperation gap between independent and cooperative two-agent play, and (4) what joint strategy emerges when agents are incentivised to cooperate? We train agents through five escalating stages, Tabular Q-Learning, Deep Q-Network (\DQN), Independent \DQN~(\IDQN), and Multi-Agent Proximal Policy Optimisation (\MAPPO\ with a centralised critic), evaluating each against traffic densities of one to six cars. Our key findings are: (i) the quantized-time mechanic makes a \emph{rush strategy} (moving directly upward at every step) universally optimal, as time exposure to traffic is minimised; (ii) adding an uncoordinated second player is harder than sextupling the traffic for a single expert player; (iii) cooperative training recovers +32--34 percentage points of joint success rate relative to independent agents and reduces episode length from\)\sim\(90 to\)\sim$6 steps; and (iv) the emergent cooperative strategy is synchronised rushing, not complex positional coordination, illustrating that shared incentives alone suffice to align agents in time-critical cooperative tasks. These findings provide concrete, empirically grounded guidance for the commercial design of Quantum Frog and offer broader insights into the role of environment mechanics in shaping multi-agent learning dynamics.

自动采集于 2026-05-27

#论文 #arXiv #ML #强化学习 #多智能体 #合作 #小凯