Searching for Harmful Refusal: A Psychometric Audit of an AI Safety Attribute in HELM Safety

Forum topic · 小凯 · 2026-10-11

Summary

This arXiv paper (2610.12409) by Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, and Hoda Heidari examines whether "harmful refusal"—a model's tendency to refuse dangerous or policy-violating prompts—constitutes a single, measurable attribute in the HELM Safety benchmark. Using a construct validity framework, the authors analyze HELM Safety's four candidate datasets and find three are saturated. Psychometric tests on HarmBench show that a multidimensional item response theory model strongly indicates HarmBench does not measure a single attribute, and differential item functioning analysis reveals that models from different developers with equal refusal ability score differently on some items. These warnings largely disappear under domain-specific matching, consistent with aggregation effects but not ruling out developer-specific differences. The paper argues that averaging across datasets and items can hide saturation and confound behaviors, and that any safety score should prove worthy of a single-attribute interpretation before being used to compare models.

Paper Overview

Research Area: Machine Learning Authors: Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, Hoda Heidari Published: 2026-10-08 arXiv: 2610.12409

Abstract (English)

Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles. Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute. One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts. We examine whether it constitutes a single, measurable attribute in HELM Safety. Using a construct validity framework that stipulates that an attribute must exist before a test can measure it, we start with HELM Safety's four datasets that might plausibly target harmful refusal, but find three saturated. We subject the remaining dataset, HarmBench, to two psychometric tests to determine whether a single attribute such as harmful refusal is likely to underlie its scores. First, a multidimensional item response theory model strongly suggests HarmBench does not measure a single attribute. Second, a differential item functioning analysis finds that models from different developers with equal refusal ability score differently on certain items. These cautions largely disappear under domain-specific matching, a pattern consistent with aggregation effects but insufficient to rule out domain-specific developer differences. Zooming out, HarmBench folds distinct harmful behaviors into one score, and HELM Safety's overall score further folds HarmBench's score together with other datasets' into a top-line number. Any safety score averaged across datasets and items may hide saturation and confound behaviors in this way. We argue that before a score is used to compare models, it should first demonstrate that it merits a single-attribute interpretation.

Key Takeaways

Overall benchmark scores can mask very different safety attribute profiles across models.
Of HELM Safety's four candidate datasets for harmful refusal, three are saturated; only HarmBench remains analyzable.
Multidimensional IRT indicates HarmBench does not measure a single attribute; DIF analysis reveals developer-specific item effects.
The authors call for construct validity checks before interpreting any safety score as a single attribute.