Summary
This paper formalizes benchmark-free comparative safety scoring for large language models, addressing deployments that must compare candidate models before labeled benchmarks exist in a target language, domain, or regulatory regime. The authors define a validity contract for scenario-based audits: scores are only valid under fixed scenario packages, rubrics, auditors, judge models, sampling configurations, and re-run budgets. Lacking ground truth, they substitute a tool validity chain consisting of responsiveness to controlled safety-vs-ablated contrasts, dominance of target-driven variance over auditor and judge artifacts, and stability across re-runs. The chain is instantiated in SimpleAudit, a local-first scoring tool, and validated on a Norwegian safety package: safety and ablated targets separate with AUROC 0.89-1.00, target identity is the dominant variance component (eta-squared about 0.52), and severity distributions stabilize over ten re-runs. Applying the same chain to Petri shows it accommodates multiple tools. A Norwegian public-sector procurement case comparing Borealis and Gemma 3 illustrates that the safer model depends on scenario category and risk metric, so scores, matched differences, critical ratios, uncertainty, and the auditors and judge models used must be reported together rather than collapsed into a single ranking.
Paper Overview
Field: Machine Learning
Authors: Sushant Gautam, Finn Schwall, Annika Willoch Olstad, Fernando Vallecillos Ruiz et al.
Published: 2026-05-07
arXiv: 2605.06652
Summary
Many deployments must compare the safety of candidate language models before labeled benchmarks exist for the relevant language, domain, or regulatory regime. The paper formalizes this setting as benchmark-free comparative safety scoring and makes explicit the contract under which scenario-based audits can be interpreted as deployment evidence.
Key Ideas
- Validity contract: Scores are only valid under a fixed scenario package, scoring rubric, auditors, judge models, sampling configuration, and re-run budget.
- Tool validity chain instead of ground truth: Since no labels are available, the authors replace agreement-with-truth with a chain of checks:
1. Responsiveness to controlled safety-vs-ablated contrasts
2. Dominance of target-driven variance over auditor and judge-model artifacts
3. Stability across re-runsValidation in SimpleAudit
The chain is instantiated in SimpleAudit, a local-first scoring tool, and validated on a Norwegian safety package:
- Safety vs. ablated targets separated with AUROC values of 0.89 to 1.00
- Target identity was the dominant variance component (eta^2 ~ 0.52)
- Severity distributions stabilized over ten re-runs
Applying the same chain to
Petri shows that the approach accommodates multiple tools; substantive differences arise upstream of the chain, in declared-contract enforcement and deployment adaptation.
Procurement Case Study
A Norwegian public-sector procurement case comparing Borealis and Gemma 3 demonstrates the resulting evidence in practice: which model is safer depends on the scenario category and the risk metric.
Reporting Recommendation
Scores, matched differences, critical ratios, uncertainty estimates, and the auditors and judge models used must be reported together, rather than collapsed into a single ranking.
---
*Auto-collected on 2026-05-10*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177619695