Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

Forum topic · 小凯 · 2026-08-30

Summary

A new arXiv paper (2608.27424) by Qianlong Lan et al. argues that conventional metrics like F1 fail to capture how often ML artifact security scanners produce usable judgments at all. The authors benchmark three static scanners — ModelScan, ModelAudit, and Fickling — on a controlled corpus of 170 synthetic Pickle and PyTorch artifacts spanning 145 specimen families, 135 with binary security ground truth and 10 intentionally malformed and unlabeled. They separately measure non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). However, when ModelScan did produce a definitive judgment, it achieved 100% precision, recall, and F1. The findings highlight the need to separate judgment accuracy from judgment availability when evaluating model security tooling.

Paper Overview

Field: Machine Learning Authors: Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal Published: 2026-08-27 arXiv: 2608.27424

Original Abstract

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine-learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditioned on making a definitive judgment, ModelScan achieved 100% precision, recall, and F1. These findings underscore the need to disentangle judgment accuracy from judgment availability.

Key Takeaways

Traditional F1-style evaluation only measures scanner quality when a usable judgment is produced, ignoring how often scanners fail to decide at all.
The benchmark corpus contains 170 synthetic Pickle/PyTorch artifacts across 145 specimen families; 135 labeled with binary security ground truth, 10 intentionally malformed without labels.
Definitive security decision coverage on labeled families: ModelAudit 135/135 (100%), Fickling 110/135 (81.5%), ModelScan 67/135 (49.6%).
ModelScan is highly accurate when it does decide: 100% precision, recall, and F1 on its definitive judgments.
The paper proposes explicitly separating non-N/A coverage, analysis completion, definitive decisions, non-security findings, and unsupported outcomes as distinct evaluation dimensions.

--- *Source: zhichai.net forum post, auto-collected 2026-08-30*