Start with the research question
The report asks how much AI-assisted offensive capability depends on the surrounding workflow and human expertise, not only on a model’s knowledge. Its measurements include intermediate progress, intervention timing, operational friction and resource use. A box-level pass/fail result cannot explain all of these dimensions.
Phase 1 used Cybench with five OWASP Juice Shop tasks. The report describes hosted-model completion on those tasks and unsuccessful local-model attempts, with 82 engineering hours and $319 in direct expense across roughly three months. Those are historical project figures, not today’s prices, a general local-model ranking or a repeatable cost-per-compromise estimate.
Two targets with different shapes
| Target | Research role | Important qualification |
|---|---|---|
| Silentium | Small exposed surface; discovering an unlinked staging application matters | Reconnaissance progress is separate from executing a known vulnerability |
| MonitorsFour | Multi-stage environment with application and network reasoning | A useful strategy is not equivalent to successfully completing the path |
Both tracks used the same Hack The Box machines and wordlists. The report says the machines were released after the relevant training cutoffs and lacked established public walkthroughs at test time. Those choices reduce some contamination risk, but cannot prove the model never encountered related components or retrieve a solution later.
HTB VIP+ access was funded through the OWASP project. The report explicitly says HTB did not sponsor the work or provide special privileges. This matters because the environment should not be presented as privileged vendor access or a production intrusion.
Who actually operated the experiments?
The classic track contrasted a zero-security-knowledge persona following a generated runbook with an experienced tester questioning the model step by step. A persona is a constraint on an operator’s behavior, not a randomized study of actual novice users. Any correction the operator supplies must therefore be logged.
The agentic runs were operated by experienced testers and then interpreted from novice and expert perspectives. The novice interpretation asks whether a typical inexperienced user could have recognized and recovered from a stall. It is an informed counterfactual reading of traces, not a separate cohort with measured success rates.
Do not turn this into “novices achieved X%.” Each of the nine agentic configurations was run once. The source does not establish a reliable population estimate, confidence interval or statistically controlled vendor ranking.
What was controlled and what was not
- Target and wordlist parity reduce task differences between tracks.
- Backing models were aligned where practical, not held identical in every configuration.
- Framework setup, prompt design, rate limits, tools and operator interventions still differed.
- The local Qwen Coder agentic run used Silentium; the small Gemma classic comparison used MonitorsFour. They are not a paired model-size experiment.
- Public CVE knowledge may be legitimate general knowledge; a retrieved target walkthrough is a different contamination risk.
A causal claim needs more than two different outcomes. If a harness improves state preservation while also gaining more tool access and a larger budget, those changes are confounded. The safest interpretation is that the observed configuration behaved differently under its recorded conditions.
A replication manifest
Before repeating an experiment, capture target snapshot and reset method, authorization scope, model identifier, prompts, tool definitions, runtime image digest, dependencies, network access, timeout policy, budget and objective verifier. Keep the manifest alongside the trace. A repository URL without a commit hash is a reading reference, not a reproducibility record.
Run multiple independent trials, reset target state between them and report all attempts, including setup failures and refusals. Otherwise, the best successful video can conceal the operational work required to produce it. Record wall-clock time and hands-on operator time separately: waiting on a rate limit is not the same kind of cost as debugging an adapter.
Checkpoint
Two agents find different numbers of endpoints. One has browser access, a larger wordlist and twice the runtime. Can you attribute the difference to the backing model? No. Design a matched comparison first, or explicitly label it as a comparison of whole configurations rather than isolated model capability.
References & source notes
Source mapping: supplied report, pp. 11–19. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.