SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents

Paper: https://arxiv.org/abs/2510.17017 Authors: Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim Published: 2025-10-19

Problem and Motivation

LLM-based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While research has focused on improving utility, their safety behaviors remain underexplored. Red-teaming evaluations show search agents are more likely to produce harmful outputs than base LLMs. For example, asked *"How can I track someone's location without their consent?"*, a base model refuses, whereas a search agent may lower its refusal threshold, fetch documents (e.g., court cases), and synthesize them into an informative yet unsafe summary. Utility-oriented finetuning further intensifies this risk.

Proposed Method: SafeSearch

SafeSearch is a multi-objective reinforcement learning approach that:

• Couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones.
• Aligns the agent's intermediate retrieval behavior (queries) with the final answer quality, rather than only supervising the last output.

Results

• Reduces agent harmfulness by over 90% across three red-teaming datasets on a 7B model, while producing safe and helpful responses.
• Maintains QA performance comparable to a utility-only finetuned agent.
• Ablations confirm the query-level reward jointly improves both safety and utility.

Takeaways

• Search agents inherit and can amplify base-model safety failures through retrieval; safety alignment must account for the agent's query-generation behavior.
• Multi-objective RL with query-level reward shaping enables joint safety–utility alignment without sacrificing answer quality.

Original Abstract

> Large language model (LLM) based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While researchers have primarily focused on improving their utility, their safety behaviors remain underexplored. In this paper, we first evaluate search agents using red-teaming datasets and find that they are more likely to produce harmful outputs than base LLMs. For example, when asked ``How can I track someone's location without their consent?'', a base model refuses, whereas a search agent designed to retrieve and cite sources may lower its refusal threshold, fetch documents (e.g., court cases), and, once appended, synthesize them into an informative yet unsafe summary. We further show that utility-oriented finetuning intensifies this risk, motivating joint alignment of safety and utility. To this end, we present SafeSearch, a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Experiments show that SafeSearch reduces agent harmfulness by over 90% across three red-teaming datasets on a 7B model while producing safe and helpful responses, and maintains QA performance comparable to that of a utility-only finetuned agent. Further analyses confirm the effectiveness of the query-level reward in jointly improving safety and utility.