[论文] Imitation Bootstrapped Multi-Agent Reinforcement Learning
研究领域: ML 作者: Tony H. Zhang, Connor Mattson, Daniel S. Brown 发布时间: 2026-11-13 arXiv: 2611.18543
论文概要
研究领域: ML 作者: Tony H. Zhang, Connor Mattson, Daniel S. Brown 发布时间: 2026-11-13 arXiv: 2611.18543
中文摘要
演示是加速单智能体强化学习(RL)的有效方法,但将这一思路扩展到多智能体强化学习(MARL)并非易事,因为一个人很难同时远程操作多个智能体。轮询行为克隆(R2BC)通过让单个演示者一次控制一个智能体来解决这个问题,而其他智能体执行其当前策略。然而,得到的模仿策略受限于演示的质量。我们提出模仿引导的多智能体强化学习(IBMARL),一个将模仿引导强化学习(IBRL)扩展到合作团队的框架,并使用R2BC来训练模仿策略。在每个时间步,每个智能体从其模仿策略和RL策略各接收一个动作提案,一个集中式评论家集成在两种策略的每智能体组合中进行选择。门控模仿惩罚——一种软Q过滤的行为克隆损失——仅在评论家认为模仿动作更优时将每个RL策略正则化向其模仿策略,使得训练结束后每个智能体单独执行其RL策略,无需集中式评论家。在四个合作多智能体RL基准上,IBMARL在两项任务上比基于演示的基线快2.6-4.2倍超越其模仿策略,在另外两项任务上是唯一实现超越的方法,并在四项中的两项上超越了演示者本身。使用真实人类演示时,IBMARL仍能超越单纯模仿学习,且速度是最佳基线的两倍多。
原文摘要
Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives...
*自动采集于 2026-11-17*
#论文 #arXiv #ML #小凯