The primary reference
This course adapts LLM Exploit Generation: Phase 2: Agentic vs. Classic Offensive Approaches, version 1.0, October 2026, from the OWASP GenAI Security Project’s Threat Intelligence Initiative. The user supplied a 57-page PDF. Page references throughout this course refer to that supplied edition; an online revision may paginate differently.
Open the official OWASP release page ↗. The report is a source for study observations; this adaptation does not claim to have rerun its experiments. External incidents and benchmark figures in its bibliography are not automatically independently established by repeating them here.
Attribution and ShareAlike
The report credits contributors Ella Duffy, Andrea Delgado Fukushima, Rachel James, Bryan Nakayama, Ben Ramirez, Lily Reynolds and Tom Stripling; reviewers Christine Bryan and Mayur Agnihotri. We credit the OWASP GenAI Security Project and the contributors listed in the supplied edition. No OWASP endorsement of OverWatch Labs or this adaptation is implied.
The source states that it is licensed under Creative Commons Attribution-ShareAlike 4.0 International. The adapted educational text in this GenAI section is distributed under the same licence. Attribution, the licence link and an indication of changes must accompany redistribution. Existing site branding, unrelated pages and third-party software remain subject to their own terms.
Changes by OverWatch Labs: reorganized into web chapters; paraphrased explanatory text; added a glossary, evidence taxonomy, synthetic lab exercises, reference catalogue, navigation and engineering checklists; made limitations and uncertain tool identities explicit. The lab examples and recommended design patterns are teaching additions, not results measured in the original experiment.
Read claims at the right confidence level
- Report observation: an outcome or trace described by the authors. We attribute it to the report.
- Primary upstream: a project, provider document or research paper used as a reading reference. Its existence is not verification of every downstream claim.
- Editorial recommendation: our proposed engineering or teaching pattern, clearly distinct from a measured improvement.
- Unresolved reference: an ambiguous identity or a link whose content could not be independently confirmed. Do not infer that the claim is false or verified from reachability alone.
The bibliography includes reports of evaluation incidents, model capability assessments and measurement critiques. This course does not rely on those incidents as proof of a universal capability claim. Follow the primary source, check its date and methodology, and distinguish an allegation, preliminary assessment and reproduced observation.
Complete report bibliography
The report lists 47 works cited and seven works consulted. Links checked 28 October 2026: reachability and title checks are not replication of research.
- Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard Link checked 28 Oct 2026
- When AI Benchmarks Plateau Link checked 28 Oct 2026
- Anatomy of a Frontier Lab Agent Intrusion: July 2026 timeline Link checked 28 Oct 2026
- Assessing Claude Mythos Preview’s Cybersecurity Capabilities Link checked 28 Oct 2026
- CrowdStrike 2025 Global Threat Report findings Link checked 28 Oct 2026
- CrowdStrike 2026 Global Threat Report findings Link checked 28 Oct 2026
- AIRTBench Link checked 28 Oct 2026
- Detecting and Countering Misuse of AI: August 2025 Link checked 28 Oct 2026
- Eval Awareness in Claude Opus 4.6’s BrowseComp Performance Link checked 28 Oct 2026
- Frontier Safety Framework v3.1 Link checked 28 Oct 2026
- Gemini 3 Pro Frontier Safety Framework report Link checked 28 Oct 2026
- AutoPenBench Link checked 28 Oct 2026
- Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access Link checked 28 Oct 2026
- GPT-5.3-Codex System Card Link checked 28 Oct 2026
- GPT-5.5 System Card Link checked 28 Oct 2026
- Inside Tycoon2FA Link checked 28 Oct 2026
- Investigating Three Incidents in Our Cybersecurity Evaluations Link checked 28 Oct 2026
- Holistic Agent Leaderboard Link checked 28 Oct 2026
- Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks Link checked 28 Oct 2026
- CTFusion Link checked 28 Oct 2026
- LLM Cyber Evaluations Don’t Capture Real-World Risk Link checked 28 Oct 2026
- On Leakage of Code Generation Evaluation Datasets Link checked 28 Oct 2026
- When Agents Disagree With Themselves Link checked 28 Oct 2026
- METR / Redwood: OpenAI and Hugging Face incident investigation Link checked 28 Oct 2026
- OpenAI and Hugging Face model-evaluation incident Primary page checked in browser 28 Oct 2026
- Offensive Cyber Task Horizons Link checked 28 Oct 2026
- OpenAI Preparedness Framework v2 Link checked 28 Oct 2026
- Anthropic Responsible Scaling Policy Link checked 28 Oct 2026
- Hugging Face security incident disclosure: July 2026 Link checked 28 Oct 2026
- NYU CTF Bench Link checked 28 Oct 2026
- CyberGym-E2E Link checked 28 Oct 2026
- RAND: AI Agents Put Offensive Cyber within Reach of Novices Primary page checked in browser 28 Oct 2026
- Finding Widespread Cheating on Popular Agent Benchmarks Link checked 28 Oct 2026
- Claude Opus 4.6 System Card Link checked 28 Oct 2026
- Claude Opus 4.8 System Card Link checked 28 Oct 2026
- Hacking CTFs with Plain Agents Link checked 28 Oct 2026
- UK AISI / CAISI preliminary assessment of Kimi K3 Link checked 28 Oct 2026
- CyberGym Link checked 28 Oct 2026
- ExploitGym Link checked 28 Oct 2026
- Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities Primary page checked in browser 28 Oct 2026
- PentestEval Link checked 28 Oct 2026
- Rethinking Benchmark and Contamination with Rephrased Samples Link checked 28 Oct 2026
- Harness-Bench Link checked 28 Oct 2026
- Cybench Link checked 28 Oct 2026
- Stop Comparing LLM Agents Without Disclosing the Harness Link checked 28 Oct 2026
- CVE-Bench Link checked 28 Oct 2026
- CTF-Dojo Link checked 28 Oct 2026
- 3CB (consulted) Link checked 28 Oct 2026
- Measuring What Matters: Construct Validity (consulted) Link checked 28 Oct 2026
- Are Mythos’ Cyber Capabilities Overhyped? (consulted) Link checked 28 Oct 2026
- METR Frontier Risk Report (consulted) Link checked 28 Oct 2026
- AI Agents That Matter (consulted) Link checked 28 Oct 2026
- BountyBench (consulted) Link checked 28 Oct 2026
- Establishing Best Practices for Building Rigorous Agentic Benchmarks (consulted) Link checked 28 Oct 2026
Editorial limitations
The supplied report contains forward references to a Phase 3 or Section 10 agenda without a corresponding full Section 10 in its contents. We do not invent that missing material. Some tool names and checkpoints lack identifiers; those gaps are labelled in the catalogue. Model and repository states can change after the reference check date.
A reachable URL does not certify title accuracy, experimental validity or software safety. Automated checks are useful for dead links, but should not be marketed as independent replication. Keep this distinction when sharing a chapter, citing a result or choosing a system for a real assessment.
References & source notes
Source mapping: supplied report, pp. 2, 51–55. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.