3CB: the catastrophic cyber capabilities benchmark

Abstract

Introduces the Catastrophic Cyber Capabilities Benchmark (3CB), a framework for assessing offensive cyber capabilities of LLM-based agents, organized around the MITRE ATT&CK technique taxonomy. The benchmark contains 15 original challenges spanning 14 attack strategies and demonstrates that frontier models such as GPT-4o and Claude 3.5 Sonnet can already carry out reconnaissance, exploitation, and lateral-movement tasks, exposing significant gaps in existing cybersecurity evaluations for AI systems. – AI-generated abstract.