How to use this catalogue
This catalogue includes tested frameworks, surveyed systems, operating environments, command-line tools, target applications, model families and benchmarks named in the report. Each entry explains its role so “mentioned” is never mistaken for “tested.” PyRIT is included as supplementary context for the Microsoft agent, not as an additional Phase 2 experiment.
Prefer the upstream documentation or repository over a third-party tutorial. For benchmarks, a primary paper is linked when the PDF does not identify a unique implementation. A link identifies a project; it does not pin a release or establish compatibility with your environment. Review licenses and maintenance state before using anything.
Unresolved names remain visible. Expo has no confirmed matching upstream. Mythic and Kali-GPT have explicitly labelled candidate links. The exact Nmap/Metasploit MCP bridges and Qwen checkpoint are not specified by the PDF; we do not invent them.
Browse the references
69 entries. Unresolved identities are labelled below.
Agent frameworks
CAI
Tested Alias Robotics harness. Upstream now marks this repository archived; historical results are not a maintenance recommendation.
Open primary reference Link checked 28 Oct 2026HexStrike AI
Tested MCP tool server orchestrated by Claude Code or Codex.
Open primary reference Link checked 28 Oct 2026Claude Code
Tested standalone and in a bespoke multi-agent configuration.
Open primary reference Link checked 28 Oct 2026OpenAI Codex
Tested standalone and as an orchestrator. Repository identifies the CLI, not historical model availability.
Open primary reference Link checked 28 Oct 2026Surveyed systems
PentestGPT
Surveyed, not one of the four tested framework families.
Open primary reference Link checked 28 Oct 2026Mythic
IDENTITY UNRESOLVED: candidate Mythic C2 upstream only. The PDF provides no repository or version, so this is not a confirmed match.
Candidate upstream (unconfirmed match) Link checked 28 Oct 2026Kali-GPT
IDENTITY UNRESOLVED: candidate upstream only; multiple independent projects use this name. Not official Kali Linux.
Candidate upstream (unconfirmed match) Link checked 28 Oct 2026Expo
IDENTITY UNRESOLVED: no unambiguous security-framework upstream established. Deliberately not linked to the unrelated mobile framework.
No confirmed upstream linkMicrosoft AI Red Teaming Agent
Official product reference. Evaluates AI systems; not interchangeable with infrastructure penetration testing. Exact surveyed release unspecified.
Open primary reference Link checked 28 Oct 2026PyRIT
Supplementary reference: Microsoft identifies this as underlying its red-teaming agent, not separately tested in Phase 2.
Open primary reference Link checked 28 Oct 2026Orchestration and MCP
LangChain
Used in early prototypes, not a separately scored final configuration.
Open primary reference Link checked 28 Oct 2026LiteLLM
Provider abstraction associated with CAI; compatibility and rate limits remain separate concerns.
Open primary reference Link checked 28 Oct 2026Model Context Protocol (MCP)
Tool/context protocol, not by itself an authorization boundary.
Open primary reference Link checked 28 Oct 2026Burp Suite MCP server
Official PortSwigger bridge; alternative build path discussed in the PDF.
Open primary reference Link checked 28 Oct 2026Nmap MCP integrations
BRIDGE UNSPECIFIED: PDF names no bridge repository. Link identifies underlying Nmap only.
Open primary reference Link checked 28 Oct 2026Metasploit MCP integrations
BRIDGE UNSPECIFIED: PDF names no bridge repository. Link identifies the underlying framework only.
Open primary reference Link checked 28 Oct 2026Security tools
Nmap
Network discovery and service inspection. ICMP response is not equivalent to reachability.
Open primary reference Link checked 28 Oct 2026curl
HTTP inspection. Record request context and redirect chain with the response.
Open primary reference Link checked 28 Oct 2026Gobuster
Directory/virtual-host discovery; wildcard responses need negative controls.
Open primary reference Link checked 28 Oct 2026ffuf
HTTP fuzzing utility, named in framework troubleshooting.
Open primary reference Link checked 28 Oct 2026Metasploit Framework
Modular testing framework; a proposed module is not verified execution.
Open primary reference Link checked 28 Oct 2026Burp Suite
Web-security testing suite, distinct from its MCP extension.
Open primary reference Link checked 28 Oct 2026SecLists
Wordlist collection. Pin list contents, not just a conventional path.
Open primary reference Link checked 28 Oct 2026DIRB
Official Kali package reference; the PDF discusses a missing dirb wordlist.
Open primary reference Link checked 28 Oct 2026Lab infrastructure
Kali Linux / kali-rolling
Distribution used in the lab. VM and minimal-container contents differ.
Open primary reference Link checked 28 Oct 2026Podman
Container runtime; network capability assumptions caused classic-track failures.
Open primary reference Link checked 28 Oct 2026OpenVPN
VPN connection to the isolated lab environment.
Open primary reference Link checked 28 Oct 2026SSH / OpenSSH
Protocol/client reference. Exact Python SSH library is not pinned in the PDF.
Open primary reference Link checked 28 Oct 2026Python
Scripting runtime. Generated imports and APIs need independent checks.
Open primary reference Link checked 28 Oct 2026Node.js
Runtime reference; capture versions and dependency state.
Open primary reference Link checked 28 Oct 2026Next.js
Application-framework reference; a fingerprint alone does not establish a vulnerability.
Open primary reference Link checked 28 Oct 2026Models and providers
Ollama
Local serving for the CAI/Qwen Coder run; record model artifact and quantization.
Open primary reference Link checked 28 Oct 2026ChatGPT / GPT models
Family reference for GPT-4o and GPT-5.5 labels in the PDF. Not a claim that every historical variant is available today.
Open primary reference Link checked 28 Oct 2026Claude models
Provider reference for Claude 3.5 Sonnet and Opus 4.5/4.6/4.8 labels. Exact historical model IDs remain necessary for replication.
Open primary reference Link checked 28 Oct 2026DeepSeek-R1
Phase 1 family; distinguish distilled checkpoint, size and quantization.
Open primary reference Link checked 28 Oct 2026Gemma 3 / 270M
Official Google model card for the size named in the classic comparison; the study does not pin its exact quantized artifact. Not representative of all local models.
Open primary reference Primary page checked in browser 28 Oct 2026Qwen Coder
CHECKPOINT UNSPECIFIED: family-level upstream, not proof that Qwen3-Coder was the checkpoint tested.
Open primary reference Link checked 28 Oct 2026Gemini / Gemini 3 Pro
Provider reference; discussed in the bibliography, not a tested Phase 2 framework.
Open primary reference Link checked 28 Oct 2026Kimi / Kimi K3
Provider reference; K3 appears in a cited assessment, not the nine configurations.
Open primary reference Link checked 28 Oct 2026Claude Mythos Preview
Source-cited assessment; distinct from the ambiguous Mythic framework.
Open primary reference Link checked 28 Oct 2026Targets and applications
Hack The Box
Silentium and MonitorsFour are study targets; target access and walkthrough availability change.
Open primary reference Link checked 28 Oct 2026OWASP Juice Shop
Intentionally vulnerable application used in Phase 1 and cited evaluations.
Open primary reference Link checked 28 Oct 2026Flowise
Silentium staging application; version/configuration must establish CVE applicability.
Open primary reference Link checked 28 Oct 2026Cacti
MonitorsFour application; locating a public PoC is not verified execution.
Open primary reference Link checked 28 Oct 2026Microsoft HTTP Server API
Reference for the server banner discussed in classic-track diagnosis.
Open primary reference Link checked 28 Oct 2026Defensive references
MITRE ATT&CK
Behavior taxonomy. Mapping a technique does not prove AI attribution.
Open primary reference Link checked 28 Oct 2026OWASP AISVS
Verification standard; check edition before interpreting control identifiers such as C9.2.
Open primary reference Link checked 28 Oct 2026Threat actors’ use of AI register
Community evidence register cited by the PDF. Assess underlying incident sources independently.
Open primary reference Link checked 28 Oct 2026Benchmarks
Cybench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026CVE-Bench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026AIRTBench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026InterCode / InterCode-CTF
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026NYU CTF Bench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026CTFusion
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026CTF-Dojo
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026AutoPenBench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026CyberGym
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026CyberGym-E2E
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026ExploitGym
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026PentestEval
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026Harness-Bench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 20263CB
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026BountyBench
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026Holistic Agent Leaderboard (HAL)
Primary research paper. A paper link is used where the report does not pin an implementation. Tasks, harnesses and scoring rules differ; scores are not interchangeable.
Open primary reference Link checked 28 Oct 2026SWE-bench Verified
Coding benchmark discussed as a measurement example, not an offensive-cyber result.
Open primary reference Link checked 28 Oct 2026BrowseComp
Official evaluation repository; discussed in relation to evaluation awareness.
Open primary reference Link checked 28 Oct 2026Model names are not reproducibility records
The report names GPT-4o, GPT-5.5 Thinking/Pro/Codex, Claude 3.5 Sonnet, Opus 4.5/4.6/4.8, DeepSeek-R1 variants, Gemma 3 270M and Qwen Coder. It also cites Gemini 3 Pro, Kimi K3 and Claude Mythos Preview assessments. Provider-family links are deliberately used where an exact historical checkpoint is absent.
Record the exact identifier returned by the provider or serving system, not only a marketing label. For local models, also record artifact hash, architecture, quantization, tokenizer and inference settings. For agent tools, record client, server, adapter and schema versions separately. Availability and defaults may change after publication.
What is not counted as a tool
The PDF ends with sponsor and supporter organizations. Those acknowledgements are not a technical inventory or a claim that their products were evaluated. We preserve the source attribution without presenting every sponsor as a tested security system. The full research bibliography is separate from this catalogue.
References & source notes
Source mapping: supplied report, pp. 6–19, 23–34, 41–55. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.