Agentic Workflow Resilience Audit: 70-Point Checklist

An agentic workflow resilience audit grades a production agent stack against the failure modes that actually take it down — timeouts that never fire, retries without idempotency, rollback paths that don't roll back, human checkpoints placed where they don't help. The 70 checks below are the seven-axis grid we run on client engagements before any agent workflow goes near real traffic.

The pattern is consistent across audits. Capability is rarely the problem in 2026 — the models are strong enough. What separates a prototype that demos cleanly from a system that survives a production weekend is the resilience layer: bounded execution, safe retry, compensating actions, surgical human review, and a trace good enough to replay the incident afterwards. Every team with an agent in production has been bitten by at least one of those.

This guide walks through what each axis covers, the most common findings we see, and a worked example applied to a 3-stage research agent so you can run the audit on your own workflows today. Skip to the FAQ for the questions teams ask before booking a formal audit.

The pattern is so common it's almost a cliché. An engineer wires together three or four agent steps in a notebook, gets a clean demo, ships it behind a webhook, and moves on. Three weeks later the first incident lands: an upstream API rate-limits, the agent retries forever, a Slack channel fills with duplicate notifications, and someone manually kills the process. The fix is never "use a better model" — it's the resilience layer that was never built.

Resilience is a discipline, not a feature. It shows up as bounded execution time, safe retry semantics, explicit compensation paths, surgical human review at the steps that actually need it, and traces good enough to walk through the failure after the fact. The 70-point checklist below maps to six axes that compose the discipline.

Timeouts are the cheapest resilience primitive and the most commonly missing one. A workflow that runs forever is worse than one that fails — the failed workflow surfaces the bug; the infinite workflow just burns money. The audit grades timeout coverage at three scopes, and the failure mode is almost always the same: timeouts exist at one scope but not the others.

The ten checks

• End-to-end workflow timeout. Every workflow has a wall-clock bound. Hit it, the workflow terminates with a defined error.
• Per-stage timeout.Each logical step in the workflow has its own bound, smaller than the workflow timeout and matched to the step's expected latency profile.
• Per-tool-call timeout. Every external tool invocation (HTTP, database, MCP server) has a timeout shorter than the stage timeout. No tool call can hang the stage.
• LLM streaming timeout. Streaming responses have an inter-token timeout, not just a total-response timeout. A model that stalls mid-stream is detected and aborted.
• Connection vs response timeouts split. Connection establishment is bounded separately from response waiting — the two failure modes need different remediation.
• Timeouts are propagated, not absolute. If a caller has 30 seconds left, the callee gets 30 seconds minus a budget, not its own fresh 60-second timeout.
• Cleanup on timeout. Timed-out operations release resources, close connections, and emit a structured cancellation event upstream.
• Timeout granularity matches step latency. A 30-second timeout on a step whose p50 is 25 seconds will fire constantly on normal traffic. Match timeouts to p99 + buffer, not p50.
• Timeouts are configurable per environment. Dev/staging/prod have different latency floors; hard-coded timeouts cause environment-specific incidents.
• Timeout firing is alerted, not silent.Every timeout fires a structured event with stage + tool + duration; the alert threshold is "rate-of-timeouts crosses N over M minutes", not "single timeout".

The remediation pattern is mechanical. Pick the three scopes (workflow, stage, tool), set timeouts at p99 latency plus a 25-50% buffer, propagate the remaining budget down the call stack, and wire every timeout firing into the trace. Most teams can move from a 0/10 to an 8/10 on this axis in a single sprint — it's the highest-leverage axis in the audit.

Retries are where good intentions turn into incidents. A naïve retry loop against a flaky API generates a thundering herd; a non-idempotent tool call that retries on a network blip charges the customer twice. The audit grades whether retries are safe, bounded, and scoped to the failure classes they actually help with.

The ten checks

• Retry only on transient failures. 5xx, network timeouts, rate-limit signals. Never on 4xx semantic errors.
• Exponential backoff with jitter. Base delay doubles per attempt; jitter prevents synchronized retries from identical agents.
• Bounded retry attempts. A hard cap (typically 3-5) per logical operation. After the cap, escalate — do not loop.
• Idempotency keys on every mutating call. Stable key per logical step = workflow_id + step_id + input_hash. The tool dedupes server-side; the agent retries without fear.
• Retry-After header respected. If the upstream specifies a wait, the agent waits that long. Never retry before the indicated time.
• Retry budget per workflow.Total retries across all stages capped per workflow run. Prevents a degraded dependency from consuming the workflow's entire latency budget on retries.
• Per-tool retry policy, not blanket. Search queries can retry aggressively; payment calls cannot retry the same way. Policy lives next to the tool definition.
• Circuit breakers on persistent failure. If a tool fails N times in M minutes, stop retrying entirely for a cooldown window. Open circuit = explicit error to caller.
• Retry attempt logged with context. Each retry attempt emits a structured event: attempt number, prior error, backoff delay. Aggregate to monitor retry rates per tool.
• Retry policy is unit-tested. A test forces transient failures and asserts exponential backoff, attempt count, and final error propagation. Otherwise the policy rots.

Idempotency is the keystone. Without it, every retry on a mutating call is a coin flip on whether the customer gets two of whatever you just sent. With it, retries become free — the tool sees the same key, returns the previous result, and the workflow keeps moving. Generate keys at the agent layer (deterministic, content-addressed), enforce them at the tool layer (server-side cache with TTL).

Multi-step agent workflows have the same problem as distributed transactions: when step five fails, the side effects of steps one through four are still out there in the world. Rollback in the database sense doesn't work — you can't un-send an email, un-charge a card, or un-post a webhook. The saga pattern is the production-grade answer: every forward action has a registered compensating action that undoes its effect, and a failure triggers compensation in reverse order.

The ten checks

• Every mutating step has a compensating action. Send-email pairs with send-correction-email; charge-card pairs with refund-card. Registered at the step definition, not discovered at incident time.
• Compensation order is reverse-of-forward.If forward order was A → B → C and C fails, compensation runs C' → B' → A'.
• Compensation is idempotent. Same idempotency discipline as forward actions — compensation can be retried safely.
• Compensation has its own timeout and retry policy. Failed compensation is a high-severity event; treat it as such.
• Compensation failure escalates to human. If a compensation step exhausts its retries, the workflow surfaces the failed undo to an operator queue with full context.
• Irreversible steps are marked. Some actions genuinely cannot be undone (a message posted to a public channel, an external API with no reversal). Mark them, gate them behind a checkpoint, and never attempt fake compensation.
• Compensation logs are append-only. The compensation history is itself part of the workflow trace — never overwritten, always queryable for incident review.
• Compensation can run partially. If steps 1-3 succeeded and 4 failed, compensation runs only for 1-3 — not for the never-executed 4.
• End-to-end compensation drill once per quarter. A test workflow forces failure at each step and verifies compensation completes successfully.
• Compensation logic is co-located with forward. Same file, same review, same tests. Drift between forward and compensation is the most common saga failure mode.

The biggest mistake we see on this axis is not the absence of rollback — it's the assumption that database-style rollback covers external side effects. It doesn't. A workflow that sends an email at step 3, charges a card at step 4, and fails at step 5 doesn't need rollback; it needs a registered apology-email and a registered refund. Saga is more work to write up front and the only honest recovery posture once you've shipped to real users.

Human-in-the-loop is the resilience axis most teams over- or under-invest in. Too many checkpoints and the agent stops being useful; too few and the agent ships a high-blast-radius action that needed a second pair of eyes. The discipline is surgical placement — checkpoints exist only where the blast radius justifies them, with explicit timeouts and escalation paths so a stalled checkpoint doesn't stall the workflow.

The ten checks

• Checkpoints at high-blast-radius steps. Irreversible mutations, high-cost branches, sensitive customer communications. Not at every step.
• Confidence-gated checkpoints.If the agent's confidence score falls below a threshold, escalate even on normally auto-approved steps.
• Approval timeout with default action.If no human responds within N hours, the workflow takes a defined default — fail-safe (abort + compensate) or fail-forward (proceed) per step policy. Never "wait forever".
• Approver routing is deterministic. Each checkpoint type routes to a defined queue with documented SLAs. No ad-hoc DMs.
• Approval context is complete.The approver sees the inputs, the agent's plan, the proposed action, and the compensation plan. Approving with partial info is a configurable error.
• Approval is auditable. Who approved what at what time with what context — append-only log, queryable for compliance.
• Escalation paths defined per failure class. Rate-limit trip = on-call engineer; compensation failure = on-call + product owner; ambiguous customer intent = customer success queue.
• Approval UX matches urgency. Low-urgency checkpoints in an email/Slack queue; high-urgency in a paging system. Mismatched UX is how SLAs get missed.
• Checkpoint count is measured. Total checkpoints per workflow tracked over time; trending up means the workflow is regressing toward manual operation, trending down means automation is winning.
• Override paths exist for emergencies. An operator can force-fail, force-approve, or force-compensate a stuck workflow with full audit logging. No frozen state.

This is the longest axis in the audit for a reason: the difference between a 30-minute incident response and a 4-hour one is almost entirely whether you can replay the failed workflow against the captured inputs. Observability is the table-stakes layer (you can see what happened); deterministic replay is the production-grade layer (you can re-run it locally with the same inputs and prove the fix).

The fifteen checks

• Workflow trace per run. Every workflow invocation has a single trace ID linking every step, tool call, and decision.
• Tool inputs and outputs captured. Every tool call records its full input and output (with PII redaction). No guessing what the model saw.
• LLM prompts and completions logged.Including system prompt, user messages, and the model's full response. Token counts attached.
• Decision boundaries are explicit. Every routing decision (retry vs fail, escalate vs auto, compensate vs proceed) logs the inputs that drove it.
• Latency captured per stage. Wall-clock duration of each step, broken down into tool latency vs LLM latency vs agent overhead.
• Cost captured per stage. Token spend per LLM call plus tool-cost markers. Aggregable per workflow, per tenant, per agent.
• Error categorization. Errors carry a structured code (transient/permanent, retryable/not, internal/external). Free-text errors only as a fallback.
• Traces are sampled, not dropped. 100% capture at low volume; head-based sampling at scale; tail-based sampling for errors so every incident has full fidelity.
• Replay harness exists. Re-run a stored trace against the same inputs; reproduce the exact failure locally. Without this, incident response is hypothesis-driven.
• Replay uses captured tool responses. The harness stubs external tools with the responses captured in the trace — deterministic, no external dependencies, fast.
• Replay handles non-determinism. Captured random seeds, captured timestamps, captured LLM responses. Replay is byte-identical or flagged as drift.
• Per-tenant observability.Traces are filterable by tenant for multi-tenant workflows; one tenant's incident doesn't require scanning all traces.
• Alerting on aggregate signals. Retry rate per tool, timeout rate per stage, compensation rate per workflow — all alertable, not just per-incident.
• Trace retention policy. Defined retention per data class (error traces longer than success traces); audit regulator-relevant traces longer still.
• Replay is rehearsed.The on-call rotation has run a replay against a real incident at least once. Otherwise the harness rots and stops working when it's needed.

The replay capability is what unlocks confident change. Without it, every fix is a guess and every rollout is a leap of faith. With it, the incident review produces a test case, the test case becomes a regression check, and the next incident teaches the system something durable. For broader context on the observability primitives that make this possible, our agent observability audit (60-point checklist) covers the trace and metric layer in dedicated depth.

To make the framework concrete, here is the audit applied to a real-shape research agent we saw on a client engagement in February 2026. The workflow has three stages: fetch a set of source documents via a web-search tool, summarize each document via an LLM call, and synthesize the summaries into a final briefing emailed to a customer. Every step looks innocent in isolation; the resilience gaps stack into a real incident risk.

The workflow under audit

Stage 1 calls a search API with a customer-supplied query, returns 10-25 URLs. Stage 2 fetches each URL, extracts the body, sends it to an LLM for a structured summary; runs in parallel up to 5 at a time. Stage 3 takes the summaries, calls an LLM for synthesis, then calls an email-send tool to deliver the briefing to the customer. A single workflow run touches roughly 30-50 LLM calls and exactly one mutating side effect (the email).

The initial audit score: 18 / 70

The workflow had been in production for six weeks when the audit ran. The score breakdown was depressingly typical:

• Timeouts: 3 / 10. An overall workflow timeout existed (15 minutes) but no per-stage or per-tool timeouts. The workflow had stalled twice in the prior month on a slow upstream search API.
• Retries: 4 / 10. Exponential backoff was wired around the LLM calls but not around the search API or the email send. No idempotency key on the email — meaning a retry risked sending duplicates.
• Rollback: 1 / 10. No compensation for the email step. If the email sent but the workflow failed post-send, the customer received an incomplete briefing with no follow-up.
• Human-in-the-loop: 2 / 10. No checkpoints anywhere. The email — a customer-facing irreversible action — sent automatically with no human review and no confidence gate.
• Observability + Replay: 8 / 15. Reasonable tracing of tool I/O; no replay harness. Incident debugging required reading raw logs.

The remediation plan

We sequenced the fixes by leverage, not by axis order — fixing the highest-blast-radius gaps first, then filling in the rest:

• Sprint 1 (timeouts and email idempotency). Add per-stage timeouts: 60s for search, 90s/document for summary with a 5-concurrent cap, 120s for synthesis, 30s for email send. Add an idempotency key on the email-send tool (workflow_id + customer_id + briefing_hash). Estimated effort: one week. Score lift: 18 → 38.
• Sprint 2 (saga + HITL on the email step). Register a compensating "send-correction-email" action against the email step. Add a confidence-gated human checkpoint before the email send: if the briefing's synthesis confidence falls below a threshold, route to a reviewer queue with a 4-hour timeout and fail-safe default (don't send). Estimated effort: one week. Score lift: 38 → 54.
• Sprint 3 (replay + alerting). Build a replay harness that re-runs a stored trace with stubbed tool responses. Wire aggregate alerts on retry rate per tool, timeout rate per stage, compensation rate per workflow. Rehearse one replay end-to-end on a historical incident. Estimated effort: one week. Score lift: 54 → 64.

The result

Three weeks of focused work moved the workflow from 18/70 to 64/70 — Stage 1 (happy-path) to Stage 4 (hardened product). The measurable change post-audit: incidents per month dropped from 3 to 0 over the following quarter, and the one near-incident in month four was caught and contained by the new confidence gate before it reached the customer. The audit cost roughly four hours of senior engineer time; the remediation cost three engineer- sprints. Most teams will see similar ratios — the audit is cheap, the remediation is the investment.

If you want the same audit applied to your agent workflows, our AI transformation engagements include resilience audits as a standard line item; we've also published companion audits for agent observability and the subagent design discipline that frequently sits underneath these workflows.