KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux

Forum topic · 小凯 · 2026-10-03

Summary

KaliBench (arXiv:2610.02206) is a fine-grained benchmark and dataset for evaluating how well large language models translate natural-language requests into executable CLI commands for real-world cybersecurity tools on Kali Linux. It contains 8,504 query-command pairs covering 1,642 tools across 23 capability dimensions and 5 security phases. Unlike existing evaluations that focus on knowledge tests or end-to-end agentic tasks, KaliBench directly measures executable command generation, using a documentation-based pipeline with deterministic normalization and alias-aware evaluation to assess tool selection and argument construction precisely and reproducibly. A multi-stage validation pipeline combining LLM verification, sandboxed terminal execution, and human refinement ensures semantic correctness and real executability. The benchmark also supports runtime-free verifiable-reward training. Across three evaluation modes and 24 configurations, no open-source model exceeds 42% exact-command accuracy in unrestricted settings, highlighting the difficulty of using CLI security tools without in-prompt hints. Supervised fine-tuning and reinforcement learning with verifiable rewards on KaliBench substantially improve 8B models, bringing them to performance comparable to a 685B MoE model.

Paper Overview

Field: NLP
Authors: Pengfei Li, Naufal Suryanto, Sicheng Zhang, Muzammal Naseer
Published: 2026-10-01
arXiv: 2610.02206

Motivation

LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks, and do not directly measure LLMs' ability to generate executable commands for real-world cybersecurity tools. This gap is critical because cybersecurity operations rely on strict command-line interfaces (CLIs), where minor syntax errors, incorrect flag-value bindings, or argument misordering can invalidate execution.

What Is KaliBench?

KaliBench is a fine-grained benchmark and dataset for natural-language-to-CLI translation on Kali Linux:

8,504 query-command pairs
1,642 tools
23 capability dimensions
5 security phases

The dataset is built via a documentation-based pipeline with deterministic normalization and alias-aware evaluation, enabling precise and reproducible assessment of tool selection and argument construction.

Validation Pipeline

To ensure semantic correctness and real-world executability, the authors developed a multi-stage validation pipeline combining:

1. LLM verification 2. Sandboxed terminal execution 3. Human refinement

Key Findings

Based on these fine-grained deterministic signals, KaliBench supports runtime-free verifiable-reward training.
Across three evaluation modes and 24 configurations, no open-source model surpasses 42% exact-command accuracy in unrestricted settings, highlighting the difficulty of accurately using CLI security tools without tool hints in prompts.
Supervised fine-tuning and reinforcement learning with verifiable rewards on KaliBench significantly improve an 8B model, bringing it to performance comparable to a 685B MoE model.