Define the capability you are measuring
A benchmark is an operational definition of a task, not capability itself. Solving a packaged CTF, producing a patch, choosing a plausible exploit and completing an authorized end-to-end assessment require different evidence. A score becomes misleading when its label implies more than its success condition measures.
The source surveys Cybench , CVE-Bench , AutoPenBench , CyberGym , CyberGym-E2E , ExploitGym , PentestEval and other evaluations. Their targets, hints, stages, budgets and validators differ. Use the benchmark catalogue to reach the primary papers, then read the actual task definition before comparing published percentages.
Report several outcomes, not one score
| Metric | Definition | Common trap |
|---|---|---|
| Verified completion | Predefined objective confirmed independently | Counting a proposed exploit as a result |
| Assisted completion | Objective completed with human input | Hiding expert recovery inside “autonomous” |
| Stage progress | Verified intermediate milestones | Treating all milestones as equally difficult |
| Intervention burden | Setup, steering and finishing logged separately | Counting only keyboard time during the run |
| Efficiency | Evidence gained per time or total cost | Mistaking many tool calls for useful coverage |
| Failure reason | Budget, environment, interface, policy or task failure | Collapsing every stop into inability |
A refused action and a failed exploit can both prevent completion, but have different meanings. Report both task outcome and terminal reason. Similarly, a successful human-finished run is useful evidence of assistance, but cannot be placed in an autonomous-success numerator.
What pass@k does and does not say
Pass@1 concerns a single attempt under the stated protocol. Pass@k asks whether at least one of k sampled attempts succeeds. More attempts consume resources and can increase the chance of a success, so disclose k and the aggregate budget. Never compare pass@1 and pass@k as though they describe the same operating conditions.
Under an idealized independent-attempt model:
P(at least one success in k attempts) = 1 - (1 - p)^k
This is an illustration, not an estimate from Phase 2.
Shared failures and correlated attempts violate independence.A practical study also needs repeated trials, uncertainty estimates, reproducible seeds where meaningful, target resets and a record of all failed attempts. Nine configurations run once are nine case observations, not nine stable probabilities.
Separate contamination, saturation and gaming
- Contamination: prior exposure to the task, solution or privileged artifact undermines the intended test.
- Saturation: tasks become too easy to distinguish stronger systems, even when no leakage occurred.
- Gaming: a system satisfies the scoring mechanism without demonstrating the intended capability.
- Evaluation awareness: behavior changes because the system recognizes it is being tested; the consequences depend on what changes.
Post-cutoff targets reduce one contamination path, but browsing can reintroduce public walkthroughs. Log retrieval, limit prohibited sources and test whether target-side success checks can be bypassed. A contamination audit should document what it checked; absence of a published audit is not proof of contamination.
Do not infer live capability from unlike tasks
The report discusses gaps between staged and end-to-end results across several external studies. Those comparisons motivate caution; they do not establish a controlled universal multiplier between lab performance and real-world intrusion. Different environments impose different ambiguity, access, recovery and safety requirements.
Report a task’s starting information, granted credentials, available tools, allowed network, time budget and objective. State whether the system had to discover the route itself or was given a vulnerability description. These details often matter more than a model family name in the headline.
Checkpoint
A vendor reports 80% success on a staged benchmark, while another reports 10% end-to-end success with no hints. Which is stronger? The numbers alone cannot answer. First align task definitions, budgets, validators and assistance. Then compare matched results, or clearly state that no direct comparison is justified.
References & source notes
Source mapping: supplied report, pp. 37–44. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.
Adapted by OverWatch Labs from the OWASP GenAI Security Project’s LLM Exploit Generation: Phase 2 (October 2026). Reorganized, paraphrased and supplemented with teaching examples. No OWASP endorsement implied.