GitHub’s Outage Postmortem: Client Retries Made It Worse

GitHub’s August 17 outage postmortem, published on August 20, 2026, does something day-of status updates never can: it names the mechanism. In GitHub’s words, the outage began when traffic reached a new peak and a critical infrastructure component in its Central US data center failed to scale with it. Then, as those services threw errors, a client-side retry loop increased traffic during recovery, and GitHub had to mitigate that behavior before it could safely restore traffic. The outage lasted 7 hours and 47 minutes.

That second sentence is the one worth reading twice. A capacity failure is a vendor problem. A retry loop that makes recovery harder is a client problem, and every team that runs CI bots, webhook consumers, agent fleets or orchestrator retry wrappers against GitHub is a client. GitHub’s first listed remediation uses the phrase “retry budgets” and “retry storms” directly, which means the vocabulary for fixing this is GitHub’s own, not something we are importing.

This post covers what the postmortem establishes and what it deliberately leaves generic, where its figures differ from our own day-of tracking, the scale of the remediation, Copilot as a separate failure domain, and then a retry-budget standard grounded in the AWS Architecture Blog, a second Amazon engineering write-up and Google’s SRE book, with an audit checklist you can run against your own harness this week.

The post is titled “The August 17 outage, and the work ahead” and it opens without hedging: “On August 17, GitHub experienced an outage that lasted 7 hours and 47 minutes.” The same opening names every affected surface in one sentence: “It disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot, affecting developers and organizations around the world.” GitHub also frames the event as “our second significant incident in August, following an actions failure on August 6,” a detail we note as context only.

The root-cause paragraph is short and specific about the where, and deliberately generic about the what: “Our investigation found that the outage began when traffic reached a new peak, and a critical infrastructure component in our Central US data center failed to scale with it.” No component is named. The next two sentences are the heart of this post.

Three things follow from that wording. First, the retry loop is described as a recovery problem, not the trigger: capacity failed first, retries made the way back harder. Second, GitHub says it had to mitigate the loop before it could safely restore traffic, which means the retry behavior was on the critical path to recovery, not a side effect. Third, GitHub does not say which client was retrying, and it keeps that abstraction throughout the post. We keep it too.

The tone is unusually direct for a company postmortem. “If you were trying to ship software that day, we let you down,” GitHub writes, and the closing line commits to earning trust “through the scaling and reliability of the platform.” Whether that lands depends on the remediation list in section 04, which is where the numbers are.

The mechanism GitHub describes is the textbook cascading-failure shape. A backend starts returning errors because it cannot scale. Clients see the errors and retry. Each retry is a fresh request against a backend that is already at or over capacity, so the error rate rises, so more clients retry, and the load the backend must absorb to recover keeps climbing precisely because it has not recovered. Google’s SRE book calls this out directly: without randomization, “a small perturbation (e.g., a network blip) can cause retry ripples to schedule at the same time, which can then amplify themselves.”

GitHub’s own incident summary, which we read via its Community Discussions mirror of the status-page record, gives one concrete measure of how far that amplification went. The Copilot Token Service normally serves 7,000 to 9,000 requests per second. During the incident it saw 70,000 to 100,000 requests per second. Taking the ends of both ranges, that is between roughly 8 and 14 times normal, which is to say an order of magnitude of extra traffic arriving at a service that was already struggling. We use that figure because it is GitHub’s, it is about a named internal service rather than a named client, and it illustrates amplification without asserting who caused it.

That abstraction matters for the lesson. If the post had named a single client, the temptation would be to file this as that client’s bug. Because it did not, the useful reading is the general one: any client that retries a failing request without a budget is a potential amplifier, and the only defense a platform has against an unbudgeted client is to throttle it, which is what “mitigate that behavior” means in practice. For the client-side half of the story, our API error handling and resilience reference covers the retriable-versus-non-retriable distinction that every retry policy has to start from.

On August 17 we published GitHub Came Back Before Copilot Did, built from the 36 updates on GitHub’s status-page incident record. That post recorded the incident opening at 13:40:03 UTC and resolving at 21:15:46 UTC, for 7 hours 36 minutes, and stated that no root cause had been published. The August 20 postmortem gives a different duration, 7 hours and 47 minutes, and GitHub’s incident summary puts the start at 13:28 UTC with the same 21:15 UTC close. We are presenting both rather than silently overwriting our own published figure, because the difference is itself the story of what a retrospective adds.

The two start times are 12 minutes apart and the two durations are 11 minutes apart. The most likely reading, and it is our inference rather than anything either source states, is that the two figures measure from different start points: the moment the status page was first marked as investigating versus the moment GitHub’s own telemetry says traffic crossed the new peak. We are not going further than that. Both sources that give an end timestamp agree on 21:15 UTC. One further delta sits in the table: our day-of post recorded Actions mitigated at 16:59 UTC and staying clear, while GitHub’s incident summary puts an Actions degradation resolving around 18:03 UTC. We present both with their sources, as with the duration.

The larger lesson is about what each kind of source is for. A live-tracked timeline tells you when the vendor knew and what it said; a retrospective tells you what actually happened and when it started. Both are legitimate, and an incident review on your side should keep both columns rather than backfilling the retrospective number over the day-of one. The agentic workflow incident-response playbook has a runbook template that keeps those columns separate by design.

The remediation section is where the postmortem stops being a narrative and becomes a capacity plan. One sentence carries the headline figures: “We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity.” Those are two clauses of a single sentence, not separate initiatives, and GitHub gives no breakdown of where the cores went.

The second figure is the Azure migration, and it is stated as two metrics in one breath: “Today, Azure serves roughly 58% of GitHub’s platform load and half of all Git operations, up from 12% of platform load in May.” GitHub gives no date within May for the 12% baseline and no date for “today” beyond the August 20 publish date, so both ends should be read as approximate. Taken at face value, that is a move of roughly 46 percentage points of platform load across a May-to-August window. GitHub does not break that shift down by month and does not say the migration changed pace, so nothing in the pair of figures isolates anything that happened after August 17. The signal is the size of the move, not its timing.

Then come the engineering changes, and the order matters. First: “we are applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions to prevent retry storms and cascading load.” Second: “we are reviewing lower-priority CPU and memory alerts to identify components that could fail during sudden traffic spikes.” The post also describes architecture work toward “unlimited read operations,” rolling out gradually beginning with the largest monorepos, which is framed as ongoing rather than complete.

Note the scope of that first item. GitHub is applying retry budgets to its own service-to-service calls, which is the internal half of the problem. The external half, the clients that retry against GitHub, is the half only clients can fix, and it is the half the rest of this post is about.

Our day-of post argued, from the outside, that the AI assistant in a developer loop sits on a different dependency chain from git hosting and recovers on a different clock. The postmortem confirms it from the inside in one sentence: “Most GitHub services recovered earlier that day, but some Copilot services took longer.” GitHub’s incident summary adds the detail that the Copilot Token Service, the service whose request rate spiked by an order of magnitude, was fully recovered by 21:02 UTC, 13 minutes before the incident closed at 21:15 UTC, while the status-page record had shown the Central US data center recovering by 16:36 UTC. We are not re-telling the outage day here; the timeline, the 16:59 UTC mitigation list and the relapses are all in the August 17 post. What the two GitHub sources add together is that the service which took longest to recover was also the one whose request rate spiked by an order of magnitude. GitHub does not connect those two facts explicitly, and neither do we beyond noting them side by side.

“Retry budget” has a precise meaning, and it predates this outage by a decade. Google’s SRE book describes two distinct budgets. In the cascading-failures chapter it is process-wide: “Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don’t retry; just fail the request.” In the overload chapter it is a per-client ratio: “Each client keeps track of the ratio of requests that correspond to retries. A request will only be retried as long as this ratio is below 10%.” The payoff of the ratio budget is stated directly: “layering on the per-client retry budget (a 10% retry ratio) reduces the growth to just 1.1x in the general case.”

Backoff shape comes from the AWS Architecture Blog’s “Exponential Backoff and Jitter”, Marc Brooker’s 2015 post that is still the canonical reference for the technique. It compares three jitter formulas and finds that Full Jitter did the least total client work of the three, because in a simulation of 100 contending clients plain exponential backoff without jitter produced synchronized retry spikes, and jitter spread those out to an approximately constant rate with less total client work. The post does not prescribe a cap or a maximum attempt count; those come from the SRE references. A separate Amazon write-up, “Timeouts, retries, and backoff with jitter”, supplies the rule for where retries belong: in a five-deep stack of service calls with three retries at each layer, it says the load on the database at the bottom increases 243 times, “making it unlikely to ever recover,” which is why Amazon states that “for low-cost control-plane and data-plane operations, our best practice is to retry at a single point in the stack.” The table below combines those four references with what GitHub’s own postmortem and REST API documentation say.

The three jitter formulas are short enough to show in full, and they are the part teams most often get wrong by implementing plain exponential backoff and calling it done.

None of this is new, and that is the point. The references are a 2015 blog post, a continuously maintained book and Amazon’s own write-up on timeouts, retries and backoff. GitHub’s postmortem is a reminder that a platform at record scale still had to add consistent retry limits and retry budgets to its own internal calls in August 2026, which makes it unlikely your harness has them either. Our API rate-limiting reference covers the server side of the same contract.

To be precise about what follows: GitHub does not identify the clients behind the August 17 retry loop, and nothing in its post suggests AI coding agents were involved. This section is about what comes next, not what happened. The class of client that retries on a 5xx has grown. A year ago it was CI runners, webhook consumers and a handful of integrations. Today it also includes agent fleets that clone, push, open pull requests and poll Actions on their own schedule, orchestrators that wrap those agents in their own retry logic, and tool-calling loops that treat a failed API call as something to try again immediately.

That last layer is where multi-layer amplification lives. Google’s SRE book warns that when retries happen at multiple levels, “a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer.” An orchestrator that retries a task three times, wrapping a sub-agent that retries its tool call three times, wrapping an HTTP client that retries three times, is multiplicative by construction. The fix is structural: retry at one layer, and make every other layer fail fast and report.

GitHub’s REST API documentation already tells clients how to behave when the platform pushes back. Authenticated requests get 5,000 per hour, unauthenticated 60 per hour, and a GitHub Actions GITHUB_TOKEN gets 1,000 per hour per repository; secondary limits cap concurrency at 100 requests and content-generating requests at 80 per minute. On a 403 or 429, the documented behavior is to wait the number of seconds in the retry-after header if present, to wait until the x-ratelimit-reset timestamp if x-ratelimit-remaining is zero, to wait at least one minute when neither applies, and to use exponential backoff for continued failures. The docs also warn that continuing to make requests while rate-limited can result in the integration being banned. Read the rate-limits page as a contract your fleet has already agreed to. GitHub’s companion best-practices page adds two more rules: wait at least one second between each request when making a large number of POST, PATCH, PUT or DELETE calls, and, when a request keeps failing against a secondary limit, back off exponentially and “throw an error after a specific number of retries” rather than retrying indefinitely.

Looking forward, the asymmetry is the thing to plan for. A platform can add three million CPU cores; it cannot make an unbudgeted client behave. Agent fleets, CI bots, webhook consumers and orchestrator retry wrappers all retry without a person watching the counter, and every one of them needs the discipline GitHub is now applying to its own services. The teams that treat retry budgets as part of the harness, rather than as something the HTTP library probably handles, are the ones whose fleets will still be making progress during the next partial outage instead of being throttled out of it. If that harness work is on your roadmap, our AI transformation engagements start from exactly this kind of reliability audit.

Six of the seven questions below map to a row of the standard in section 06. The seventh, on idempotent writes, has no row there, because it guards the write path rather than the retry rate. Run them against CI bots, webhook consumers and orchestrator retry wrappers specifically, because those are the clients most likely to have inherited a default retry policy nobody has read. A “no” on any row is a retry-amplification risk, and the fix column names the reference the fix comes from.

For the full surface, our 70-point agentic workflow resilience audit folds these seven into a broader review, and the webhook reliability and idempotency reference goes deep on question seven. If you would rather have the audit run for you, it is a standard part of our web development engagements for teams shipping agent-driven pipelines.