KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

The 30-second take

• What: KaliBench measures whether language models can turn an analyst's request into an executable Kali Linux command, not just answer cybersecurity trivia or finish an end-to-end agent run.
• Why it matters: Security operations live or die on exact CLI syntax; a wrong flag or argument order fails the job, so a graded, runtime-free reward is a path toward cheaper, more reliable tool-using assistants.
• Who should care: Teams building defensive security copilots, evaluators of tool-using models, and anyone who wants capable command-line help without assuming a giant model is required.

What the paper actually did

The authors introduce KaliBench, a fine-grained benchmark and dataset for translating natural-language intent into Kali Linux command-line invocations. It contains 8,504 query–command pairs covering 1,642 tools across 23 capability dimensions and five security phases. Construction uses a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation so tool choice and argument construction can be scored reproducibly. A multi-stage verification stack — model-based checks, sandboxed terminal execution, and human refinement — is meant to keep pairs both semantically correct and practically executable. Those deterministic signals also support runtime-free verifiable rewards for training. Across three evaluation modes and 24 configurations of general-purpose and security-focused open-weight models, no open-weight model exceeds 42% exact-command accuracy in the unrestricted setting. Supervised fine-tuning and reinforcement learning with KaliBench-derived rewards then lift an 8B model to performance comparable to a 685B mixture-of-experts model.

What makes this disruptive

Most existing cybersecurity evaluations test knowledge or whole-agent outcomes and skip the narrow, brittle skill of writing a correct command. KaliBench isolates that skill at CLI granularity, where a single misbound flag invalidates execution. The reported unrestricted-setting ceiling — no open-weight model above 42% exact-command accuracy — is a hard public baseline, not a marketing claim. Pairing that baseline with runtime-free verifiable rewards is the second punch: the same graded signals used for scoring can train a small model toward a much larger one. That combination attacks the scarcity of reliable, specialist command-line labor without requiring a 685B-class model at inference. It is a measurement-and-training paper, not a product launch, but it changes what “good at security tools” can mean in a leaderboard.

Why it matters (outside the lab)

Abundance lens (today’s luxuries → tomorrow’s defaults): expert security command-line work is still scarce, expensive, and easy to get slightly wrong. If a shared, executable-command benchmark becomes the default way to train and audit tool-using models, capable defensive assistance can move from a few large specialist stacks toward smaller, cheaper models that still hit the syntax. Horizon: near-term for evaluation and fine-tuning practice; not a calendar promise that every analyst gets a flawless copilot. Near-term use is to update baselines and training recipes. Medium-term, cost, reliability, and independent replication decide whether this becomes ordinary infrastructure rather than a one-off 8B-versus-685B result.

Limitations & open questions

This is a preprint. The headline 42% figure is for open-weight models in an unrestricted setting; the paper does not claim every closed model fails the same way. Exact-command accuracy is a strict metric — semantically close but non-canonical commands may be useful in practice and still score as wrong. Training gains are reported for a specific 8B-versus-685B comparison and should not be read as a general law that any small model matches any large one. Sandboxed verification and human refinement reduce but do not erase dataset error. The work measures and trains command generation; it does not evaluate full incident-response judgment, organizational policy, or safe deployment. Nothing here demonetizes cybersecurity expertise on a fixed date.