Models, agents & harnesses

What “vibe hacking” means here

The report uses vibe hacking for human-guided security work in which an LLM supplies substantial planning and technical interpretation, while a person executes actions and returns observations. The human may follow a generated runbook or ask for one next step at a time. In either case, the person remains part of the execution loop.

This is a workflow description, not a new vulnerability class. It does not mean that a novice can safely paste arbitrary commands into production. The significant question is how much expertise the assistant supplies, and how much the human quietly contributes when the environment differs from the model’s assumptions.

Separate the layers

The observed system includes all four layers plus the operator. Replacing the model while changing the prompt, runtime, tools and budget does not isolate a model effect. Conversely, a tool-call parser crash does not show that the model lacks knowledge of the target.

Human-guided versus agentic execution

In a human-guided loop, the assistant proposes an action, the operator decides whether to execute it, and the operator selects which output to return. That selection can be a hidden source of expert assistance. A short “that is a wildcard redirect” correction can radically alter the next decision.

In an agentic loop, the harness can execute a permitted tool request and return structured output without a person relaying every step. The model gains speed and continuity, but environment faults, weak state management and incorrect assumptions can also repeat faster. More autonomy is therefore a change in authority and feedback, not proof of better judgment.

1. ObserveCollect bounded tool output
2. InterpretSeparate fact from hypothesis
3. AuthorizeCheck scope and permission
4. Act & verifyValidate the resulting state

Use an evidence ladder

• Proposed: a model describes a test. No action has occurred.
• Attempted: a tool request ran. Its exit status alone says little about target state.
• Observed: preserved output establishes a narrow fact, such as a returned page or authenticated session.
• Verified: an independent check establishes the predefined objective.
• Human-finished: a person completed an unfinished step. Credit that separately from autonomous success.

For example, “this version may be affected” is a hypothesis. “The application returned this version string” is an observation. Neither establishes the exact patch level, vulnerable configuration or successful execution. Recording each claim at its correct level prevents impressive narration from becoming a false result.

MCP connects tools; it does not make them safe

MCP supplies a common way for clients and servers to describe tools and exchange context. The permissions of the tool server, the validation of its inputs and the network access of its process still determine what it can do. A helpful tool description is not a sandbox, and a model’s assertion of authorization is not an access-control decision.

Treat responses from target pages, repository files and tool output as untrusted data. They can contain text that looks like an instruction to the assistant. Keep the trusted task and execution policy separate from those observations. The controls chapter explains how to enforce this outside the model.

Checkpoint

An agent cannot resolve a redirect hostname and repeatedly scans the same IP. Classify the problem before changing models. The useful first hypothesis is an environment or hostname-resolution issue, not that the target is offline. What evidence would distinguish these explanations? Preserve the redirect, resolver error and already-observed reachable service, then make one bounded diagnostic check.

References & source notes

Source mapping: supplied report, pp. 6–10, 14–15, 35–36. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.