Three Faces of a Safety Scoreboard: When "Harmful Refusal" Refuses to Be Measured by One Ruler

This post is the second installment of a Chinese forum series called "The Recursive Dilemma of Evaluation," translating and expanding on a psychometric audit of AI safety benchmarks. The series previously audited capability scoreboards (METR's time-horizon metric); this one turns to safety scoreboards — the rankings that decide whether models ship, who procures them, and what goes into model cards.

Key points

• The audited paper: "Searching for 'Harmful Refusal': A Psychometric Audit of an AI Safety Benchmark" (arXiv:2610.12409, 2026-10-08), by eight authors from CMU, Google, and Indiana University. Instead of testing models, it tests the ruler itself, applying psychometric validity theory (Borsboom et al., 2004) to ask whether the score measures what it claims.
• Target: HELM Safety's "harmfulness" score (v1.17.0, downloaded 2026-06-21, 81 models) — a composite of four datasets: HarmBench, SimpleSafetyTests, AnthropicRedTeam, and the harmful subset of XSTest.

Finding 1: Three of four datasets are saturated

• Three of the four datasets have pass rates between 0.92 and 0.94 — the items are too easy to discriminate between models (saturation).
• Only HarmBench (pass rate 0.67) differentiates models, meaning the entire leaderboard's refusal signal rests on a single dataset — and HarmBench itself mixes semantically diverse item types (direct malicious requests, contextual traps, copyright verbatim-recall requests).

Finding 2: The construct is multidimensional

• Multidimensional item response theory (MIRT) was applied to 398 items. A unidimensional 2PL model yields a held-out log-loss of 0.470; exploratory multidimensional models (2–10 dimensions) all drop to ~0.28; the strongest unidimensional 3PL reaches only 0.322.
• A confirmatory three-factor model by semantic category — standard (direct harmful requests), contextual (contextual traps), copyright (protected text recall) — achieves 0.258, with AIC/BIC favoring it; every confirmatory 3D random restart outperformed every unidimensional 3PL restart across 20 restarts per split.
• Factor correlations: standard–contextual = 0.785 (largely the same behavior), while copyright correlates only 0.451 and 0.603 with them. Copyright recall depends on memorization, not refusal judgment — a distinct construct. Removing copyright items still shows 2D beating 1D (0.278 vs 0.293), confirming three genuinely different behaviors.
• Brier score improves from 0.096 (unidimensional 3PL) to 0.078 (3D) after binarization.

Finding 3: Aggregation effects fabricate developer bias

• Differential item functioning (DIF) analysis comparing OpenAI models (21) vs Anthropic models (11): using a single total score for ability matching, the Mantel–Haenszel test flags 13 items and logistic sensitivity analysis flags 17 as developer-biased.
• After matching on 3D factor scores instead, only 1–2 items remain flagged — the apparent item-level bias was mostly an artifact of mixing behaviors into one aggregate score.
• Control condition: closed-source vs open-source models differ hugely in average score (0.776 vs 0.507), yet DIF flags zero items — the tool flags where it should and stays silent where it should, validating the method.

The verdict

• The paper's central claim: *"A score should earn its single-attribute reading before models are compared with it."* Validity evidence should precede leaderboard publication and procurement decisions, as psychometrics has required of tests for a century.
• The recommendation is not to discard HarmBench — it is the only discriminating dataset in HELM Safety — but to report three separate narrow scores (standard refusal, contextual refusal, copyright memory), each tied to its own construct.

Limitations acknowledged by the authors

• The 81 models are a convenience sample; developer groups are small and share training pipelines (not independent); item scores come from two LLM judges (gpt-4o and llama-3.1-405b), inheriting their biases and bypassing HarmBench's original classifier-based scoring; strict binarization at threshold 1.0 discards granularity; only one benchmark version was audited — though the MIRT and DIF pipeline applies to any benchmark publishing item-level responses.

Series context

The post frames a recursive dilemma: benchmarks evaluate models, psychometrics evaluates benchmarks, and the item scores themselves come from LLM judges that are also models needing evaluation. The series continues with a third installment on compliance scoreboards, where scores carry legal and regulatory consequences. Its closing message: scores should be used — but only after they have earned their interpretation.

References

1. Stewart, C. M., Botter, P., Sarabosing, N., Zhang, M., Phinnemore, R., Ghosh, S., Shen, H., & Heidari, H. *Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark*. arXiv:2610.12409, 2026-10-08. 2. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). *The Concept of Validity*. Psychological Review, 111(4), 1061–1071. 3. HELM Safety v1.17.0 data snapshot (accessed 2026-06-21; 81 models, HarmBench 398 items). 4. HarmBench (Mazeika et al., 2024) and the HELM evaluation framework (Liang et al., 2023).