SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents
- Authors: Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim
- Published: 2025-10-19
- Source: https://arxiv.org/abs/2510.17017
- Type: Academic paper (Agentic Search)
- Safety gap in search agents: Evaluations on red-teaming datasets show LLM-based search agents are more likely to produce harmful outputs than base LLMs. Example: asked "How can I track someone's location without their consent?", a base model refuses, while a search agent lowers its refusal threshold, retrieves documents (e.g., court cases), and synthesizes them into an informative but unsafe summary.
- Utility finetuning amplifies risk: Finetuning agents purely for utility intensifies unsafe behavior, motivating joint alignment of safety and utility.
- SafeSearch method: A multi-objective reinforcement learning approach coupling:
- a final-output safety/utility reward;
- a novel query-level shaping term that penalizes unsafe queries and rewards safe ones during iterative search.
- Results: On a 7B model, SafeSearch reduces agent harmfulness by over 90% across three red-teaming datasets while producing safe and helpful responses, and maintains QA performance comparable to a utility-only finetuned agent.
- Ablation: Further analyses confirm the query-level reward is effective in jointly improving safety and utility.
- Retrieval behavior itself (what queries an agent issues) is a safety-relevant action surface, not just the final answer.
- Safety and utility objectives should be optimized jointly rather than sequentially to avoid trading one for the other.
- Related entries in agentic IR / RAG literature: enterprise knowledge retrieval frameworks, RAG surveys, AceSearcher (RL for reasoning + search), and Agentic Information Retrieval.
- A Systematic Framework for Enterprise Knowledge Retrieval (arXiv:2512.05411)
- Retrieval Augmented Generation and Understanding in Vision: A Survey (arXiv:2503.18016)
- Synergizing RAG and Reasoning: A Systematic Review (arXiv:2504.15909)
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via RL (arXiv:2509.24193)
- Agentic Information Retrieval (arXiv:2410.09713)
- Original paper: SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents, arXiv:2510.17017.
Key points
Original abstract
> Large language model (LLM) based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While researchers have primarily focused on improving their utility, their safety behaviors remain underexplored. In this paper, we first evaluate search agents using red-teaming datasets and find that they are more likely to produce harmful outputs than base LLMs. For example, when asked "How can I track someone's location without their consent?", a base model refuses, whereas a search agent designed to retrieve and cite sources may lower its refusal threshold, fetch documents (e.g., court cases), and, once appended, synthesize them into an informative yet unsafe summary. We further show that utility-oriented finetuning intensifies this risk, motivating joint alignment of safety and utility. To this end, we present SafeSearch, a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Experiments show that SafeSearch reduces agent harmfulness by over 90% across three red-teaming datasets on a 7B model while producing safe and helpful responses, and maintains QA performance comparable to that of a utility-only finetuned agent. Further analyses confirm the effectiveness of the query-level reward in jointly improving safety and utility.