EpiSEM-Bench: Evaluating LLM Agent Safety and Evidence Recovery Under Missing Tool Evidence

Forum topic · 小凯 · 2026-10-24

Summary

EpiSEM-Bench is an executable benchmark for evaluating whether tool-using LLM agents remain safe when environment responses omit safety-relevant facts. Even when a user request is benign and every returned statement is true, an agent may still execute a prohibited action if a tool response hides a critical fact. The benchmark contains 300 matched unsafe/safe task pairs across six domains. By systematically suppressing key facts, it measures unsafe execution rates under missing evidence and identifies the minimum observation failure cardinality. An independent audit tool tests whether agents actively seek out missing facts, completing permissible tasks while suppressing prohibited ones. Experiments across ten models reveal substantial variation in omission robustness, evidence acquisition, and appropriate action. Giving models recovery access reduces unsafe execution rates for all ten models, but agents differ in whether they proactively obtain missing facts and use them to act correctly in both worlds. Paper: arXiv:2610.31650.

Paper Overview

Authors: Qijin Zhang, Hanzhen Wang, Yuehan Zhao, Haoyu Dai, Yao Shu, Zhongxiang Dai, Shuang Qiu
arXiv: 2610.31650

Key Points

Tool-using LLM agents rely on environmental facts returned by tools to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true.
EpiSEM-Bench is an executable benchmark for evaluating unsafe execution, safety boundaries, and evidence recovery in LLM agents under missing evidence.
The benchmark contains 300 matched unsafe/safe task pairs across six domains.
By systematically suppressing key facts, it measures the unsafe execution rate under missing critical evidence and identifies the minimum observation failure cardinality.
An independent audit tool tests whether agents proactively seek missing facts — completing permissible tasks while suppressing prohibited ones.
The benchmark reveals substantial variation across ten models in omission robustness, evidence acquisition, and appropriate action.
Recovery access reduces unsafe execution rates for all ten models, but agents differ in whether they actively obtain the missing facts and use them to take appropriate actions in both worlds.

Abstract

Tool-using language model agents rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true. EpiSEM-Bench evaluates unsafe execution, safety boundaries, and evidence recovery under missing evidence, revealing notable differences across ten models.