[论文] Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Securi...

论文概要

研究领域: ML 作者: Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal 发布时间: 2026-08-27 arXiv: 2608.27424

中文摘要

静态扫描器越来越多地用于识别机器学习工件中的可执行或其他不安全内容，但传统评估指标仅表征扫描器产生可用安全判断的情况。我们使用受控的、基于工件的基准评估ModelScan、ModelAudit和Fickling，在170个Pickle和PyTorch合成工件语料库上跨越145个样本族，其中135个有二进制安全真实标签，10个故意畸形无标签。我们明确区分非N/A覆盖率、分析完成度、明确安全决策、非安全发现和不受支持的结果。在标记族上，ModelAudit对全部135个族（100%）产生明确安全决策，Fickling对110个（81.5%），ModelScan对67个（49.6%）。在做出明确判断的条件下，ModelScan达到100%精确率、召回率和F1。这些发现强调了将判断准确性与判断可用性分离的必要性。

原文摘要

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), ...

#论文 #arXiv #ML #小凯