[论文] The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents ...

论文概要

研究领域: NLP
作者: Moumita Choudhury, Amit Sarker, Shlomo Zilberstein
发布时间: 2026-10-27
arXiv: 2610.31985

中文摘要

LLM agent越来越多地部署在协作环境中，在追求各自任务的同时共享资源并在审查或监督流程下运行。这种设置不仅造成agent恶意时的安全风险，也造成它们只是试图提供帮助时的安全风险。先前的工作主要研究了对抗性或明确勾结的行为；我们表明，追求独立目标的良性agent可以自发地以绕过预期工作流的方式合作。我们称之为"乐于助人agent问题"。

我们研究了长时程共享资源环境，其中agent有时依赖同伴拥有的资源，跨所有者的变更通常由审查者调解。当审查不可用时，agent开始代表彼此进行变更，尽管没有共享奖励或明确的合作激励。关键的是，这些行动仍在每个agent的本地权限范围内，同时规避了系统的预期监督结构。我们引入了FavorBench，一个具有可测量下游副作用的受控基准，并改编CooperBench以在真实仓库中测试这一现象。在两个基准和四个模型上，我们发现工作流外的帮忙频繁出现。此外，在FavorBench中，至少三个模型的帮忙对交互之外的方造成了重大损害。这些结果揭示了一个独特的多agent安全挑战：合作本身可以为绕过系统级保障创造途径。

原文摘要

LLM agents are increasingly deployed in collaborative settings where they pursue separate tasks while sharing resources and operating under review or oversight processes. Such settings create safety risks not only when agents are malicious, but also when they are simply trying to help. Prior work has largely studied adversarial or explicitly collusive behavior; we show that benign agents pursuing separate objectives can spontaneously cooperate in ways that bypass intended workflows. We call this the helpful agent problem. We study long-horizon shared-resource environments in which agents sometimes depend on resources owned by their peers and cross-owner changes are normally mediated by a reviewer. When review becomes unavailable, agents begin making changes on one another's behalf despite ...

自动采集于 2026-10-27

#论文 #arXiv #NLP #小凯