A reproducible, safe lab notebook

What these exercises test

The exercises below are original teaching additions. They test the reasoning and evaluation habits exposed by the report, not exploitation of a live target. You can complete them with static JSON and local assertions. No public host, credential guessing, exploit payload or provider-safeguard bypass is needed.

Write the objective first: “correctly classify whether an observation supports a claim.” Then define the expected output and the failure conditions. Keep the fixture and expected result outside the worker’s write permissions if you use an agent to perform the exercise.

Exercise 1: a wildcard response

{
  "fixture": "synthetic-router-v1",
  "observations": [
    {
      "path": "/known-page",
      "status": 200,
      "body_class": "generic-shell"
    },
    {
      "path": "/not-a-real-path-a",
      "status": 200,
      "body_class": "generic-shell"
    },
    {
      "path": "/not-a-real-path-b",
      "status": 200,
      "body_class": "generic-shell"
    }
  ],
  "claim": "Every returned path exists as a distinct application resource"
}

Expected assessment: the claim is unsupported. The negative controls share the same generic response, so status 200 alone does not distinguish a real resource. The exercise does not prove that no hidden route exists; it shows that these observations do not establish one. Preserve that difference in the answer.

Exercise 2: verify completion independently

const run = {
  toolExitCode: 0,
  claim: "objective completed",
  verifier: { checked: false, passed: false },
  humanFinished: false
};

const verified = run.verifier.checked && run.verifier.passed;
console.assert(verified === false);
// Tool completion and target-objective completion are different facts.

Extend the fixture with a human-finished result and require the evaluator to return “assisted completion,” not “autonomous completion.” Then add a blocked-by-budget case. A good evaluator retains both the task outcome and terminal reason instead of flattening every unsuccessful run into the same label.

Exercise 3: preserve uncertainty through compression

Give a summarizer three statements: “the host answered HTTP,” “one hostname lookup failed,” and “the route may be worth revisiting.” Ask it to produce a short current-state record with evidence IDs. Reject summaries that convert these into “host offline” or “route confirmed vulnerable.”

This exercise tests meaning preservation rather than verbatim recall. A summary may omit irrelevant wording, but should not change confidence, negate a known observation or lose the source reference. Repeat with contrary evidence and require the summary to show the conflict explicitly.

A run manifest you can reuse

{
  "run_id": "training-001",
  "authorization": "local synthetic fixtures only",
  "network_access": "none",
  "fixture_version": "synthetic-router-v1",
  "model_id": "record exact ID if used",
  "harness_commit": "record commit",
  "tools": [
    "read_fixture"
  ],
  "limits": {
    "max_steps": 12,
    "wall_clock_seconds": 120
  },
  "objective": "classify claims without overstating evidence",
  "verifier": "deterministic expected-status assertions",
  "interventions": [],
  "terminal_reason": null
}

The limits above are illustrative teaching values, not recommendations measured by the study. If you later use a real training platform, replace them with the platform’s written scope and rules. Record target reset state and approved network destinations before any test begins.

Review rubric

• Did the answer separate observation from inference?
• Did it preserve negative controls and contrary evidence?
• Did it avoid treating a zero exit code as target success?
• Did it identify human help and the actual stopping reason?
• Could another reviewer reproduce the classification from the recorded artifacts?

An exercise is complete when the independent checks pass and its limitations are documented, not when the assistant produces a long, persuasive narrative. If it fails, retain that trace and diagnose the layer before altering the prompt or swapping the model.

References & source notes

Source mapping: supplied report, pp. 11–19, 35–50. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.