[论文] Do AI-for-Science Agents Preserve Model Safety?

论文概要

研究领域: ML 作者: Michael Yew Chuan Lee, Manya Sriram, N. Benjamin Erichson 发布时间: 2026-10-27 arXiv: 2610.32045

中文摘要

AI-for-Science agent正越来越多地用于制定科学假设、运行自主实验和解释证据。这些agent通过将大语言模型（LLM）集成到特定领域的agent框架中来构建，这反过来可以提高LLM的科学能力并使模型能够编排科学工具。在这项工作中，我们问：当LLM被集成到AI-for-Science agent框架中时，模型安全性是否得到保持。

我们在50个有害化学请求上评估了五个agent框架。具体而言，我们研究了在直接提示、两种简单对抗提示策略和自适应攻击下安全行为如何变化。在四个最近的GPT变体上，我们的结果表明，即使在直接提示下，agent框架也会增加有害合规性。更关键的是，我们发现对基础LLM基本不成功的自适应攻击，在引入agent框架后突然变得高度成功。我们的结果表明，一旦引入框架，基础LLM的安全行为就不再持久。因此，随着AI-for-Science agent变得更加强大和自主，社区需要使用特定领域的安全基准来评估和对齐agent。

原文摘要

AI-for-Science agents are increasingly being used for formulating scientific hypotheses, running autonomous experiments, and interpreting evidence. These agents are constructed by integrating a large language model (LLM) into a domain-specific agent harness, which in turn can improve the LLM's scientific capability and enable the model to orchestrate scientific tools. In this work, we ask whether model safety is preserved when an LLM is integrated in an AI-for-Science agent harness. We evaluate five agent harnesses on 50 harmful chemistry requests. Specifically, we study how the safety behavior changes under direct prompting, two simple adversarial prompting strategies, and an adaptive attack. Across four recent GPT variants, our results show that the agent harness can increase harmful com...

#论文 #arXiv #ML #小凯