Evidence, state & independent verification

A finding is a claim with supporting evidence

Separate the task transcript from the evidence record. The transcript includes instructions, guesses and interpretations. The evidence record contains immutable observations with enough context to understand their limits. A finding points to those observations and explains the inference; it does not replace them.

At minimum, capture when and where an observation occurred, the authorized target, request identity, tool/version, input parameters, result status and a reference to the raw artifact. Redact secrets before distributing a report, but preserve restricted originals according to your retention policy. A hash helps detect later artifact changes; it does not prove the observation was truthful.

A minimal claim record

{
  "claim_id": "C-014",
  "statement": "The lab endpoint returned the signed-in user’s own record",
  "status": "observed",
  "scope": "isolated training fixture",
  "evidence_ids": [
    "E-031"
  ],
  "contrary_evidence_ids": [],
  "assumptions": [
    "fixture identity mapping is correct"
  ],
  "next_check": "independent fixture-state assertion",
  "verified_by": null
}

Use an explicit status vocabulary such as proposed, attempted, observed, verified, falsified and blocked. “Blocked” should also name the reason: permission, provider policy, budget, environment or unresolved evidence. These fields make it possible to ask whether progress is real instead of counting how many tools ran.

Make the verifier independent

A model should not be the sole judge of its own success. A lab objective might be checked by a target-side flag, a deterministic fixture assertion, a separately captured session state or a review by a second person. Define this before starting the run so the objective does not drift toward whatever the model happened to accomplish.

Independence has limits. A verifier reading only the same model-written summary is not independent evidence. A target-side assertion that the model can overwrite is not a strong boundary either. Keep verification state outside the worker’s write permissions when the exercise requires a trustworthy result.

Separate memory from authority

Use a small current-state summary for navigation, backed by an append-only event log. The summary can be regenerated; the observations remain addressable. When a summary and raw evidence disagree, resolve the discrepancy rather than letting the summary silently win.

Sharing without leaking sensitive artifacts

Public learning material should use synthetic identities and sanitized observations. Internal reports may need exact requests, but should not expose session tokens, personal data or client scope in public URLs. Keep a public explanation separate from a restricted evidence bundle. A shareable section link should point to the explanation, not an embedded credential.

For multi-agent systems, make workers reference shared evidence IDs instead of duplicating full sensitive outputs into every context. That reduces unnecessary disclosure and context growth. Access control must still be enforced by storage and tooling; asking workers not to reveal a secret is not equivalent to preventing access.

Checkpoint

A file has a SHA-256 digest and a model says it proves account takeover. What does the digest establish? Only that the bytes can later be compared for integrity. You still need to establish the artifact’s origin, identity context, relevant application state and the precise claim it supports.

References & source notes

Source mapping: supplied report, pp. 27–36, 41–50. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.