Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Forum topic · 小凯 · 2026-07-18

Summary

This arXiv paper (2607.15263) by Paul Kassianik, Blaine Nelson, and Yaron Singer argues that security-agent benchmarks should measure not only peak success under generous inference budgets but also cost efficiency in operational settings, where every reasoning step, tool call, telemetry query, and enrichment request consumes budget. The authors evaluate language-model security agents through a cost-success lens on offensive Cybench CTF challenges and defensive Splunk BOTS v1 SOC investigation tasks, comparing models at fixed cost levels and decomposing performance by inference spend versus tool spend. Findings reveal distinct scaling regimes: offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigations do not scale the same way; success depends more on disciplined tool usage, telemetry navigation, and selective enrichment than on raw reasoning budget. The authors advocate cost-aware, SOC-native evaluation that reflects economic efficiency and operational fit, showing which models are practically useful today and where defensive agents still need improvement.

Paper Overview

Field: ML Authors: Paul Kassianik, Blaine Nelson, Yaron Singer Published: 2026-07-16 arXiv: 2607.15263

Abstract

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget.

The authors evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, they compare models at fixed cost levels and decompose performance by inference spend and tool spend.

Key Findings

Distinct scaling regimes for red- and blue-team tasks: Offensive and defensive security tasks respond differently to added compute.
Offensive CTF: Performance improves with additional test-time compute; scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive.
Defensive SOC investigations: Success does not scale the same way. It depends more on disciplined tool usage, telemetry navigation, and selective enrichment than on raw inference budget.

Conclusion

The authors argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluation provides a clearer picture of which models are practically useful today and where defensive agents still need improvement.

---

*Auto-collected on 2026-07-18*