Read the configuration, not just the model label
The four tested framework families were CAI , HexStrike AI , Claude Code and OpenAI Codex . Standalone and bespoke multi-agent builds expanded these into nine configurations. The names below are the report’s historical labels. They are not current purchasing advice, an exact version manifest or a universal performance ranking.
The most important comparison is between observed progress and the human knowledge needed to continue. Setup problems, runtime steering and manual finishing are different intervention classes. A system requiring extensive expert engineering before the first command is not hands-off simply because its later tool calls are automatic.
A bounded reading of the reported runs
| Configuration | Progress or friction reported | What not to infer |
|---|---|---|
| CAI + Opus 4.8 | Partial enumeration; tool timeouts and native-CLI guidance | A timeout is not a model knowledge test |
| CAI + GPT 5.5 Pro | Partial enumeration; paid-tier and token-rate constraints | A paid account does not remove every rate limit |
| CAI + local Qwen Coder | Silentium reconnaissance; hostname loop and tool-call parsing failure | Local serving is not automatic adapter compatibility |
| HexStrike + Claude Code / Opus 4.8 | Useful discovery, but tool-selection guidance and focus drift | MCP availability does not guarantee effective use |
| HexStrike + GPT 5.5 Codex | Silentium account-takeover stage; guardrails; human-finished step | Reached a stage is not autonomous exploitation |
| Claude Code / Opus 4.8 standalone | Reconnaissance and disclosure; prolonged hash-cracking branch | Activity volume is not progress |
| GPT 5.5 Codex standalone | Account-takeover stage; guardrails and growing context | A human completion must remain separately credited |
| Bespoke multi-agent Claude / Opus 4.8 | Strong discovery; operator setup; final guardrail boundary | Parallelism is not end-to-end autonomy |
| Bespoke multi-agent Codex / GPT 5.5 | Redundant delegation needed role guidance; final guardrail boundary | More agents do not automatically improve prioritization |
On MonitorsFour, several configurations reached application credentials, administrative access or a Cacti vulnerability-research stage. Others missed an API distinction or failed to discover the next application. These are different intermediate outcomes on a different target; they should not be silently collapsed into a single success percentage.
The local-model run isolates several useful questions
The CAI/Qwen Coder attempt identified exposed SSH and HTTP, then repeatedly revisited a redirect or hostname-resolution problem. A tool-call compatibility failure eventually stopped it. The backing model, adapter and environment are all plausible contributors, and a single run cannot isolate them.
A fair follow-up would first validate the adapter with a harmless deterministic tool, then validate runtime DNS and routing outside the agent, then repeat the bounded task with the same observations available. That separates a model’s reasoning problem from a harness that cannot parse its requests.
Multi-agent design needs a division of labor
The report describes substantial setup work for bespoke multi-agent systems. In one configuration, the model initially tried to delegate an agent to every open port. That increased coordination and repetition rather than automatically improving coverage. Good delegation is tied to independent questions, not to the number of available workers.
An engineering pattern is to give a coordinator a shared evidence ledger, assign workers non-overlapping hypotheses, and use a separate verifier for claimed results. Workers should return evidence IDs and uncertainty, not only prose summaries. This is a design recommendation inspired by the failure traces, not a tested improvement measured by the report.
Why the stopping point matters
A provider refusal, token ceiling, adapter crash and exhausted hypothesis are different terminal states. Preserve that reason in the result. Do not rewrite a refusal as success because a person knew how to complete the remaining step, or rewrite an adapter failure as proof that the target was secure.
The source’s overall result is narrower and more useful than “AI can” or “AI cannot hack”: these configurations accelerated parts of an assessment, but did not demonstrate unattended final exploitation. The study did not establish the ceiling of every possible model or harness.
Checkpoint
A video ends with an agent finding a public PoC and a tester manually completing the action. How should the report describe it? “Agent-assisted discovery; human-finished validation,” with separate timestamps and evidence for each. That wording preserves genuine progress without overstating autonomy.
References & source notes
Source mapping: supplied report, pp. 14–19, 27–34. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.