How to Evaluate AI Agent Cybersecurity Capabilities

Summary

Wiz published a tutorial on building a benchmark for offensive AI agents, based on its relaunched Cyber Model Arena. The post argues that benchmarks should start from one concrete question, such as whether an agent can perform a real attack given only the task instruction, a general-purpose scaffold, a 1,000-second budget and no external knowledge sources. The full methodology and task set are available for download, and the arena v3.9 covers 26 models, four general-purpose harnesses, 59 agents and 327 challenges.

Topics

• AI agents