[论文] Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive ...

论文概要

研究领域: ML 作者: Paul Kassianik, Blaine Nelson, Yaron Singer 发布时间: 2026-07-16 arXiv: 2607.15263

中文摘要

安全智能体评估通常衡量在充裕推理预算下的峰值攻击能力，强调漏洞发现、漏洞利用开发、渗透测试和CTF完成。此类测量虽有用但不完整：在运营安全中，每一步推理、工具调用、遥测查询和富化请求都消耗预算。我们通过这一成本-成功视角，在攻击性的Cybench挑战和防御性的Splunk BOTS v1调查挑战上评估语言模型安全智能体。我们不只报告最佳成功情况，而是在固定成本水平上比较模型，并按推理支出和工具支出分解性能。我们的结果显示了红队和蓝队任务的不同扩展模式。攻击性CTF性能随额外测试时计算提升，扩展的开放权重模型可以接近前沿专有系统同时保持成本竞争力。防御性SOC调查则不以相同方式扩展：成功更依赖于规范的工具使用、遥测导航和选择性富化，而非仅依赖原始推理预算。我们认为安全智能体基准应衡量经济效率和运营适配性，以及任务成功。成本感知、SOC原生评估提供了更清晰的图景，显示哪些模型今天实际有用，以及防御性智能体仍需改进的地方。

原文摘要

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time...

--- *自动采集于 2026-07-18*

#论文 #arXiv #ML #小凯