From a compelling demo to a defensible result
An AI assistant can explain an unfamiliar service, draft a test plan and interpret a response in seconds. That is useful, but it is not the same as reliably completing a security assessment. This learning path explains the gap: what the model knows, what the surrounding software lets it do, and what the evidence actually proves.
The starting point is the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2: Agentic vs. Classic Offensive Approaches, October 2026. We have reorganized its 57 pages into a technical course, added definitions and controlled examples, and separated research observations from engineering recommendations. This is an independent OverWatch Labs adaptation, not an OWASP-endorsed training programme.
The central result: none of the nine agentic configurations in this study completed final exploitation autonomously. Several made useful intermediate progress. Some reached a route that a human subsequently completed. Keep all three outcomes separate.
Choose your reading route
| Your goal | Read first | Then apply |
|---|---|---|
| Understand the research | Foundations → Study design | Nine configurations → Evaluation |
| Build a safer testing workflow | Failure diagnosis → Evidence and state | Lab notebook → Execution controls |
| Defend against AI-assisted activity | Human-guided workflows → Operational constraints | Defender playbook → Source reading |
You do not need to memorize every framework before beginning. Start with the distinction between a proposal and an observation. If you can explain why an HTTP 200 response is not automatically a finding, you already have the right habit for reading the rest of the report.
What you should know
- Basic HTTP: requests, status codes, redirects, cookies and authentication context.
- Basic networking: IP addresses, hostnames, DNS resolution, ports and the difference between an application failure and a connectivity failure.
- Basic terminal use: exit codes, stdout/stderr, installed packages and process permissions.
- Basic security practice: written scope, authorization, evidence handling and an agreed stopping condition.
If access control is new to you, read our IDOR guide first. For choosing a testing engagement, start with pentest versus red team. The examples here are synthetic and intended for an isolated environment you own or are explicitly authorized to test; no exercise needs a public target.
What this course does and does not claim
This is not a leaderboard or a claim that one vendor is universally better. The agentic configurations were run once each, by experienced operators, against two lab machines. That makes the traces valuable for understanding failure mechanisms, but weak evidence for population-wide success rates or causal rankings.
Model names and outcomes are preserved as report labels. An exact API model ID, package release, quantization and system prompt are still needed to reproduce a run. The linked upstream may have changed since the experiment. Surveyed tools were not necessarily tested, and a link to a repository is not an endorsement.
We do not provide instructions to defeat provider safeguards or convert a refusal into an attack. Instead, we examine how to measure partial progress, keep tool authority outside the model, and stop ambiguous or out-of-scope actions. Strong security engineering does not depend on an assistant always behaving correctly.
The complete learning path
Check your understanding
A demo shows an agent finding a login page, identifying an old application version and printing a public exploit URL. Has it demonstrated exploitation? No: it has demonstrated discovery and research. A verified outcome requires an authorized action, preserved evidence and an independent success check. That distinction is the thread running through every chapter.
References & source notes
Source mapping: supplied report, pp. 6–10, 35–50. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.