Direct answer: A well-designed AI agent fails into a controlled state: risky actions stop, operators receive enough evidence to understand the incident, unfinished work moves to a safe fallback, affected outcomes are corrected, and the agent does not regain authority until the failure becomes a passing test.
Failure is broader than an outage
An agent can be online and still fail. It may produce an unsupported claim, choose the wrong tool, act on the wrong record, reveal restricted information, loop repeatedly, ignore an approval requirement, misread malicious content, or report completion when the business outcome never occurred. Reliability must cover behavior and consequences, not only uptime.
Failures can originate in the model, prompt, retrieval layer, source data, permissions, integration, workflow code, provider, or business process. Blaming the model too early delays recovery. Incident handling should reconstruct what the system saw, decided, attempted, and changed across the entire chain.
The organization should define failure before deployment. A support draft containing a small style error differs from a message sent to the wrong customer or an unauthorized refund. Severity should follow actual and potential harm, data exposure, reversibility, affected volume, and regulatory or contractual obligations.
Design the failure state before the happy path launches
Every tool and workflow needs a safe response to missing context, low confidence, timeout, duplicate request, invalid output, unavailable service, denied permission, and approval delay. “Try again” is not a complete strategy because retries can duplicate payments, messages, or record changes. Use idempotency, transaction limits, bounded retries, and explicit terminal states.
Decide which functions can degrade safely. An agent might switch from sending to drafting, from answering to collecting contact details, or from executing to opening a human review ticket. If no safe degraded mode exists, stop the workflow and communicate the interruption honestly.
| Failure condition | Safe response | Evidence to preserve |
|---|---|---|
| Source missing or contradictory | Do not infer; request clarification or escalate | Query, retrieved sources, and policy version |
| Tool timeout or uncertain result | Check system state before any retry | Request ID, parameters, response, and record state |
| Permission or approval absent | Stop the action | Identity, authorization result, and approval history |
| Unsafe or anomalous behavior | Pause affected capability and route to an operator | Full trace, configuration, model, and affected records |
Detect failures from outcomes, not fluent responses
An agent saying “done” is not proof. Verify durable system state: the meeting exists with the correct attendees, the CRM contains the expected update, the customer received the approved message, the refund has one transaction ID, or the research claims link to supporting sources. Structured tool results and downstream reconciliation expose silent failures that conversational review misses.
Monitor leading indicators such as unusual tool-call volume, repeated retries, permission denials, retrieval misses, unsupported claims, latency, token use, escalation rate, and divergence from expected action sequences. Also monitor business outcomes such as duplicate transactions, reopened cases, customer corrections, missed handoffs, and human rework.
Set alerts against a normal baseline and meaningful thresholds. Excessive noisy alerts train operators to ignore the system. Each alert should identify an owner, affected workflow, evidence link, and first containment action. High-consequence agents require faster detection and tighter limits than internal drafting tools.
Contain first and preserve evidence
When a material failure is detected, stop the affected capability without unnecessarily disabling unrelated services. Revoke or narrow tool access, pause a workflow version, disable automatic sending, lower transaction limits, or route traffic to a manual queue. If data exposure or security compromise is possible, follow the organization’s security and privacy incident process immediately.
Preserve prompts, retrieved content, user inputs, model and configuration versions, tool parameters and responses, identity and permission decisions, approvals, timestamps, provider status, and resulting record changes. Avoid editing logs or redeploying over the evidence before a snapshot exists. Determine which cases share the same configuration and failure condition.
Do not let the agent investigate itself as the only source of truth. It may help summarize evidence, but operators should validate against logs and systems of record. Establish one incident lead and a clear timeline so parallel fixes do not obscure what happened.
Correct affected work and communicate proportionately
Recovery is not complete when the software runs again. Find every affected record or person, reverse unauthorized or duplicate actions where possible, restore missing work, and place ambiguous cases in qualified review. Reconcile across systems because a partial transaction may have succeeded in one service and failed in another.
Communication should match the incident and applicable duties. Internal owners need the scope, operating impact, workaround, and next update. Customers, employees, partners, regulators, or insurers may require notice depending on harm, data, contracts, and law. Use verified facts, distinguish known from unknown, and avoid speculative reassurance.
Track correction as its own workstream with completion evidence. A repaired prompt does not resolve a customer who received incorrect information or a record changed without authority. The business owner should approve the remediation scope.
Find the control failure, not only the bad output
Root-cause analysis should ask why the system could produce and act on the failure. A hallucinated answer may reveal missing retrieval grounding, but it may also reveal that unsupported output was allowed to reach a customer. A wrong tool call may reveal ambiguous schemas, but the larger issue may be excessive permissions or absent approval.
Separate initiating cause, contributing conditions, detection gap, containment gap, and recovery gap. Then assign corrective actions to the appropriate layer: source ownership, instructions, retrieval, model, validator, tool contract, permissions, evaluation, monitoring, training, or operating procedure.
- Initiation: What input, change, outage, or attack began the incident?
- Propagation: Which permissions or missing controls allowed it to spread?
- Detection: Why did monitoring not catch it earlier?
- Impact: Which people, records, systems, and obligations were affected?
- Recovery: Which manual and technical steps restored a trustworthy state?
Turn every material failure into a release gate
Create an evaluation case that reproduces the incident and expected safe behavior. Add nearby variations so the fix is not narrowly memorized. Run the broader regression suite to ensure the correction did not break valid workflows. Validate permissions, degraded mode, alerts, and manual recovery—not only the generated answer.
Restore authority in stages. Begin offline, then draft or shadow mode, then a limited canary with tighter thresholds. A named business owner and technical owner should approve expansion based on evidence. Document accepted residual risk and update the runbook, training, and vendor record.
Review recurring patterns across incidents. Repeated retrieval failures may indicate weak content ownership; repeated approval bypasses may indicate flawed architecture; repeated human overrides may mean the task should remain assisted. The mature response is sometimes to narrow or remove autonomy rather than continue tuning.
A recovery runbook operators can execute
Prepare the runbook while the workflow is calm. It should name the on-call owner, severity levels, pause and revocation controls, evidence locations, manual fallback, reconciliation queries, communication owners, vendor contacts, restoration gates, and post-incident deadline. Exercise it with realistic scenarios, including a provider outage, malicious document, duplicate action, and wrong-customer access attempt.
Measure time to detect, contain, identify affected work, restore a safe service, and complete correction. Also measure whether the incident recurs. Recovery quality improves when operators can act without inventing policy during the incident.
- Classify severity and assign an incident lead.
- Pause affected authority and preserve evidence.
- Identify and correct every affected outcome.
- Fix the control layer and add regression evaluations.
- Restore gradually, communicate, and complete a documented review.