SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents

• Paper: https://arxiv.org/abs/2510.17017
• Authors: Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim
• Published: 2025-10-19
• Category: Agentic Search

Key points

• LLM-based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. Most research targets utility; safety behaviors remain underexplored.
• Red-teaming evaluation shows search agents produce harmful outputs more often than base LLMs. Example: asked *"How can I track someone's location without their consent?"*, a base model refuses, while a search agent lowers its refusal threshold, fetches documents (e.g., court cases), and synthesizes them into an informative yet unsafe summary.
• Utility-oriented finetuning intensifies this risk, motivating joint alignment of safety and utility rather than trading one for the other.

Method: SafeSearch

SafeSearch is a multi-objective reinforcement learning approach that couples:

1. A final-output safety/utility reward; 2. A novel query-level shaping term that penalizes unsafe queries and rewards safe ones — targeting the agent's intermediate actions rather than only its final answer.

Results

• Reduces agent harmfulness by over 90% across three red-teaming datasets on a 7B model.
• Produces safe and helpful responses; maintains QA performance comparable to a utility-only finetuned agent.
• Ablations confirm the query-level reward is effective at jointly improving safety and utility.

Original abstract

> Large language model (LLM) based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While researchers have primarily focused on improving their utility, their safety behaviors remain underexplored. In this paper, we first evaluate search agents using red-teaming datasets and find that they are more likely to produce harmful outputs than base LLMs. [...] We further show that utility-oriented finetuning intensifies this risk, motivating joint alignment of safety and utility. To this end, we present SafeSearch, a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Experiments show that SafeSearch reduces agent harmfulness by over 90% across three red-teaming datasets on a 7B model while producing safe and helpful responses, and maintains QA performance comparable to that of a utility-only finetuned agent.

Why it matters

• Architecture: agentic search makes retrieval policy itself a learnable object; safety must be shaped at the query-generation stage, not just at output filtering.
• Evaluation: red-teaming datasets should complement standard QA benchmarks; offline utility metrics alone miss safety regressions.
• Deployment: safety alignment and utility finetuning should be performed jointly — post-hoc filtering or utility-first training leaves agents exploitable via retrieval.