AI Incident Metrics: MTTR / MTTD Framework Agentic 2026

AI incident metrics are the eight-KPI framework that lets a production agent team measure how fast they detect, contain, and recover from incidents — and how the operational health of the program is trending across quarters. Mean time to detect (MTTD) and mean time to recover (MTTR) anchor the panel; severity distribution, repeat-incident rate, runbook coverage, and on-call load round it out. Without these metrics, agent ops is reactive by default — the playbook fires when something goes wrong and nobody knows whether the program is improving.

The metrics matter because agent failures compound differently from classical web incidents. A misbehaving agent can burn a week's token budget in an afternoon, cascade across thousands of runs before a single dashboard turns red, and stay invisible to latency or 5xx monitoring until the damage is done. The teams that turn agent ops from firefighting into a predictable program are the ones tracking the right indicators — and reviewing them with the same discipline classical SRE applies to availability.

This guide walks through each KPI with its formula, a recommended target band, the trend line to watch, and how it interacts with the others. It pairs with our companion incident response playbook — the playbook is the operational discipline; this framework is the measurement layer that proves it's working.

The default state of an agent program without metrics is reactive firefighting. Incidents happen, the team responds, the dashboards go green, and the program rolls on. Nobody knows whether response is getting faster or slower, whether the same failure class is repeating, whether on-call is burning out, or whether the runbooks written last quarter still match the current stack. The absence of measurement is not the absence of trouble — it's the absence of visibility into the trouble that's already present.

Classical SRE solved this for availability with a small set of durable KPIs — error budget, MTBF, MTTR, change-fail rate. Agent ops needs the same operational discipline, with the metric set adjusted for the failure surface. Latency and uptime stay relevant for the transport layer, but the metrics that actually measure agent health are detection time, recovery time, severity distribution, repeat-incident rate, runbook coverage, and on-call load. None of those are exotic; most have classical analogues. What's new is treating them as the operating dashboard for an agent program rather than an afterthought.

The framework below is the panel we install with clients before their agent workflows take real traffic. Eight KPIs, each with a formula, a target band, and a trend line. Reviewed monthly with the on-call rotation, refreshed quarterly with leadership. It replaces "how did last month go?" gut-feel with measurement that actually predicts whether the program is improving.

Mean time to detect is the elapsed wall-clock time between the moment an incident begins and the moment the team becomes aware of it. The start clock is the first datapoint that would have tripped a well-tuned detection panel — not the moment someone opens an incident ticket. The end clock is the page itself, whether automated or filed by a human noticing the symptom.

MTTD matters because on agent workflows, every minute of detection delay is roughly an order of magnitude of compounding cost on a misbehaving workflow. A retry storm caught at minute four costs hundreds of dollars; the same storm caught at hour four costs tens of thousands and may have produced poisoned outputs the team has to recall. MTTD is the single KPI most strongly predictive of MTTR — invest in it first.

We track three MTTD modes — automated detection, peer detection, and customer-reported detection — because their distribution tells the story of dashboard maturity. A healthy program catches most incidents automatically; a struggling one finds out from customers.

The MTTD trend line worth watching is the seven-day rolling median by detection mode. A program with healthy detection maturity shows automated detection trending steady or down (faster catches), peer detection trending flat, and customer-reported detection trending steadily down toward zero. The opposite shape — customer reports climbing while automated detection rate stays flat — is the classical signature of dashboard atrophy. New workflows shipped without new detection signals; the panel gradually stops covering the production surface.

For a deeper treatment of the underlying instrumentation that powers these metrics, see our companion piece on agent observability — trace coverage at tool-call granularity and per-workflow cost attribution are the prerequisites without which MTTD has nothing to draw from.

Mean time to recover is the elapsed wall-clock time from detection to full restoration of normal traffic. It is the most visible KPI in the panel and the one leadership asks about first. The mistake most teams make is reporting MTTR as a single number — "our MTTR is two and a half hours." A useful MTTR report breaks the clock down into the four phases of the incident response loop, because each phase is owned by a different discipline and each phase has a different lever to move it.

The bars below show the target distribution for a P0 incident on a healthy program. Detection plus containment together should close in under twenty minutes — the team is buying time fast. Eradication takes the bulk of the clock because diagnosis and surgical reversal are where the actual engineering work happens. Recovery is measured in tranches; the verification runbook between each one is non-negotiable.

The phase breakdown turns "our MTTR is too slow" from a blame conversation into an engineering decision. If detection is slow, invest in the panel. If containment is slow, the kill-switch isn't wired or on-call doesn't trust it enough to flip first. If eradication is slow, the runbook doesn't exist or the rollback primitives aren't first-class operations. If recovery is slow, the verification runbook is missing or the tranche schedule is undefined. Each phase has a fix; the panel surfaces which one is dragging.

We track MTTR per severity tier rather than a global average, because P0 and P3 have different operating contracts. P0 targets under 2 hours; P1 under 4 hours; P2 under 1 business day; P3 under 1 week. Averaging across severities hides whether the critical incidents are tightening or loosening.

Counting incidents is the metric leadership intuitively asks for and the metric that misleads worst. A team running ten P3 incidents a quarter is healthy; a team running two P0s a quarter is in trouble. The signal is the distribution across the four severity tiers, and the slope of that distribution over time.

The matrix below shows the operating contract for each tier — the page priority, the response shape, the MTTR target — and the distribution band that indicates a program in a healthy state. Drift toward higher severities is the leading indicator of a program in decay; drift toward lower severities is what a maturing program looks like as the detection panel catches issues earlier in their lifecycle.

Two disciplines make severity distribution useful. The first is that severity is set by the incident commander on the response call, not by the alert that pages — alerts are wrong about severity in both directions and the human on the call has authority to upgrade or downgrade. The second is that severity downgrades are explicit and logged, not silent. A P0 that becomes a P1 mid-incident gets the change announced in the response channel with the reason captured for the postmortem record. Without those two disciplines, severity becomes meaningless and every incident defaults to P0.

The trend line worth watching is severity distribution by calendar quarter. A maturing program shows P0 count flat or falling, P1 count flat, and P2 / P3 counts rising — the panel is catching issues earlier and at lower severity. A program in decay shows the opposite shape: P0 count rising, P3 count flat or falling, distribution shifting upward.

Repeat-incident rate is the percentage of incidents in a trailing window that share a root cause with an earlier incident in the same window. Formula: count of incidents where the postmortem failure class plus system-level root cause match a prior incident, divided by total incidents, evaluated on a rolling 90-day window. Target under 10%; sustained above 20% is the signal that postmortem discipline has broken down.

The metric matters because it is the cleanest single signal for whether the team is finding system causes or settling for agent-blame. A postmortem that ends at "the agent hallucinated" produces no action item that prevents the recurrence. The same failure class fires again in the next quarter, the team responds to it again, and repeat-incident rate climbs. Healthy postmortems generate concrete system-level fixes — new guardrails, new evals, new detection signals — and the same class doesn't come back.

Three operational levers move repeat-incident rate over a quarter. None of them are exotic; together they account for the bulk of programs that hold rate under 10% on a sustained basis.

The trend line worth watching is repeat-incident rate on a trailing 90-day window, plotted month-over-month. A healthy program shows rate falling toward zero as the action-item backlog clears and detection improvements compound. A program in decay shows rate rising as postmortems get shorter, action items go undone, and the same failure classes recycle. Repeat-incident rate is the metric most strongly predictive of whether next quarter's incident load gets lighter or heavier — invest in it.

Runbook coverage is the percentage of incident failure classes with a tested, current runbook. Formula: count of failure classes with a runbook last rehearsed in the prior 90 days, divided by total failure classes observed in the prior 12 months. Target 80%+; sustained below 50% is the signal that runbooks have become stale and the team is writing them under pressure during incidents.

The metric matters because runbook coverage is the leading indicator for next quarter's MTTR. A team responding to a failure class with a current runbook moves through containment and eradication in minutes; a team responding without one spends the first half-hour figuring out what the rollback even looks like. Runbook coverage today is MTTR three months from today — the lag between the two is reliable.

Five canonical runbook classes cover roughly 90% of agent incident shapes. Every production agent program should have all five wired, rehearsed quarterly, and rewritten annually.

The fifth canonical runbook is context restore — identifying the corrupted context source (RAG index, customer-data feed, system prompt template) and resetting it from a known-good snapshot. Slower than the other four classes because the team has to confirm the corruption hasn't spread before signing off on recovery; typical resolution window is 30-90 minutes.

Runbook freshness matters as much as coverage. A runbook written twelve months ago against a stack that has since changed model providers, swapped MCP servers, and migrated deploy systems is a runbook that doesn't work. The maintenance cadence is three-tiered: after every incident, the runbook is updated with anything that surprised the on-call engineer; quarterly, the team runs a tabletop drill against one rotation-chosen runbook; annually, all five canonical runbooks get a full rewrite pass against the current stack.

On-call load is the bundle of metrics that measure the human cost of running the rotation: pages per on-call week, interrupted-sleep nights per week, and weekend incident hours. Track all three. The aggregate is what predicts whether the rotation survives the next two quarters; the individual lines tell the team which lever to pull when load drifts up.

The metric matters because on-call load is the leading indicator for burnout, and burnout is the leading indicator for losing senior responders. A program that lets on-call load climb without intervention is one quarter away from a rotation collapse — the most experienced engineers leave first, the remaining team responds slower, MTTR rises, and the cycle accelerates. The cheap intervention is measurement plus a review cadence; the expensive intervention is hiring after the senior engineer has already given notice.

The trend line worth watching is on-call load by responder over a trailing 12-week window. A healthy program shows load distributed roughly evenly across the rotation; a struggling one shows two or three responders absorbing the bulk of pages while others coast — which is the precursor to the senior engineers quitting. Publish the per-responder breakdown internally so leadership can spot the imbalance before it compounds.

Three interventions move on-call load when it drifts up. First, tune alert thresholds against the false-positive log — a panel that pages on noise burns the rotation without delivering signal. Second, fill runbook gaps for the failure classes that paged on-call without one — every missing runbook is roughly an hour of unstructured triage per page. Third, schedule explicit recovery time for responders coming off heavy weeks; a day off after a 15-page rotation is cheaper than the senior engineer who quits after three of them.

If you're standing up the metrics panel from scratch and need help calibrating against your current stack, our AI transformation engagements include the eight-KPI dashboard as a standard line item — detection panels designed for your workflows, severity matrix calibrated to your business risk, on-call load tracking wired to your paging system.