Diagnose loops & operational failures

A useful failure taxonomy

“The agent got stuck” is not a diagnosis. Break the failure into the assumption that was wrong, the observation that should have challenged it, and the component responsible for recovery. A precise taxonomy makes the next experiment cheaper and prevents a model swap from masking a broken runtime.

Detect a loop by lack of new evidence

Repeated actions are not always mistakes. A retry after a documented transient error can be reasonable. A loop is more concerning when repeated requests neither reduce uncertainty nor change the target state. Compare normalized tool name, arguments, result class and evidence delta across attempts.

One proposed lab rule is to pause after several equivalent results and require a written explanation of what the next attempt will distinguish. The exact threshold is an operational choice, not a value measured by the OWASP report. A loop detector should explain its decision and allow a scoped human review, not silently invent a new strategy.

attempt_id: 0042
operation_class: http-observation
result_class: unresolved-host
new_evidence: false
repeat_of: [0038, 0040]
next_step: pause_for_environment_review

Use controls to test the premise

If every path produces a similar page, compare a known valid route with random nonexistent routes. If ICMP is silent but TCP or HTTP already worked, do not declare the host unreachable. If a command fails before sending a request, do not attribute that failure to target defenses.

The important habit is falsification: ask what observation would show your favored explanation is wrong. A negative control is chosen because it should not exhibit the claimed behavior. Preserve it with the finding so a reviewer can distinguish a real application response from generic error handling.

Prevent compaction from changing meaning

The report describes context degradation in which a tentative suggestion becomes a stronger assertion after summarization. This is an epistemic failure: the system’s memory now overstates what it knows. The fix is not simply a larger context window. It is retaining provenance and uncertainty separately from narrative summaries.

A compact record should retain claim ID, status, supporting evidence IDs, contrary evidence, last tested condition and permitted next check. A summary may shorten wording, but must not promote “candidate” to “confirmed.” If source output was truncated, record that limitation explicitly.

A recovery protocol

• Pause side-effectful operations and preserve the last successful state.
• Classify the failure using the actual error, not the assistant’s confident explanation.
• Choose one diagnostic check that distinguishes competing explanations.
• Update the claim ledger, including a falsified hypothesis when appropriate.
• Resume only inside the original scope and budget, or stop with a specific unresolved condition.

This protocol is deliberately slower than blindly issuing another command. The benefit is that each action has a reason. It also produces a trace that another analyst can review without needing to trust the model’s narration.

Checkpoint

A worker reports “authentication is impossible” after a single timeout. What should the coordinator retain? The timeout as an observation, authentication failure as an unconfirmed hypothesis, and the request context. It should not store a universal conclusion about authentication based on a transport error.

References & source notes

Source mapping: supplied report, pp. 20–36, 45–50. Read the full bibliography and editorial notes, or find a tool in the upstream reference catalogue.