← All articles

What famous outages teach about blast radius, rollouts and backups

Updated Nov 10, 2026 · 5 min read

Big outages rarely have exotic causes. A typo, a config change, a forgotten server, a backup nobody tested. What turns a small mistake into a famous one is how far it spreads and how long it takes to undo. The companies below published honest write-ups of their worst days, and together they make a short course in designing for failure.

Blameless postmortems

A postmortem is the write-up after an incident. It sets out what happened, the timeline, why it happened, and action items to make it less likely or less harmful next time. Blameless means it asks what let the mistake happen, not who made it. If one typo can take down a region, the problem is the system that allowed it. Blame teaches people to hide mistakes, and hidden mistakes come back.

Seven outages in one table

OutageWhat happenedThe lesson
Knight Capital, 2012New code reached 7 of 8 servers. A reused flag woke old code on the 8th, and removing the new code from the other 7 spread the problem. About $460 million lost in roughly 45 minutesAutomated deploys, no reused flags, kill switches
GitLab, 2017A wipe meant for a replica ran on the primary. The regular dumps had been failing, so the restore came from a snapshot about 6 hours oldTest restores, not just backups
AWS S3, 2017A mistyped input to a playbook command removed far more servers than meant, forcing two core subsystems to restartTools that refuse dangerous results
Cloudflare, 2019A firewall rule with a backtracking regular expression went to every server at once and pinned CPUs for about 27 minutesStage config like code, and bound regex run time
Fastly, 2021One customer’s valid config change hit a latent bug and 85% of the network returned errors. 95% recovered within 49 minutesFast detection, and a way to turn the trigger off
Facebook, 2021A maintenance command cut the backbone, DNS withdrew its routes, and the usual ways in were down too. About six hoursA way in that doesn’t depend on production
CrowdStrike, 2024A content update that wasn’t staged crashed Windows machines, about 8.5 million by Microsoft’s estimateStaged rollouts and validation for content too

Limit the blast radius

Blast radius is how much breaks when one thing goes wrong. The cheapest way to shrink it is a staged rollout. A change goes to a canary, a small slice of servers, first. You watch errors, CPU and crashes, then widen in steps, and roll back automatically if anything jumps.

healthy?healthy?ChangeCanary1% of serversOne region10%Everyone100%
A staged rollout. A bad change stops at the first stage and the rest never see it.

The hard part is applying it to everything. Cloudflare’s rule and CrowdStrike’s update were both config or content rather than code, the kind of change that often skips the careful path because it needs to ship fast. Both companies committed afterwards to staged rollouts for exactly that kind of change. Stages can be fast. Minutes per stage still catches a change that crashes every machine it touches.

Everyone at once5,000,000
1% first stage50,000
Machines that receive a bad update before you stop, out of 5 million. This uses a log scale, so equal steps in length mean multiplying, not adding, so bars that look close can be 1,000× apart.

Cells shrink blast radius in space instead of time. Split the system into copies that share nothing, each serving a slice of customers. A bad change or a poison request then takes down one cell, not everyone. AWS’s S3 write-up describes moving up plans to split part of S3 into smaller cells after its outage.

Make tools refuse dangerous results

The S3 engineer was authorized and following a playbook. The tool simply allowed a typo to remove too much. AWS changed it to remove capacity slowly and to refuse any change that would take a subsystem below its minimum. Facebook had an audit tool meant to block commands like the one that cut its backbone, and a bug let it through. Guard rails belong in the tools, and the tools need tests too.

Test the restore, not the backup

GitLab had backups configured. When it needed them, the regular database dumps weren’t there, because the job had been failing and its warning emails were being rejected. Recovery came from a snapshot made to copy data to staging. A scheduled restore test catches this on the first day it goes wrong. Restore a recent backup somewhere safe, check the data looks right, and alert when it doesn’t, or when there’s no new backup at all.

Keep a way back in

When Facebook’s backbone went down, the normal ways into its data centers went with it, and losing DNS broke many internal tools. Engineers had to go on site, into buildings designed to be hard to get into. If your recovery tools depend on the thing that’s broken, the outage locks you out. Keep an out-of-band path, with its own network, logins and tools, and practise using it. Facebook also noted that turning everything back on at once could cause new failures from the surge, which is why it brought services back gradually.

Make rollback safe, and keep a kill switch

Knight Capital’s new code reused a flag that once switched on old code nobody used any more. One server missed the new code, so the flag woke the old code there. Removing the new code from the other seven made all eight do the same. A rollback only helps if the old version understands everything the new one has started sending. Never give an old flag a new meaning, delete dead code, check every server is running the same version, and keep a kill switch that stops everything at once.

Mitigate first, fix later

During an incident, the first job is to stop the damage, not to understand it. Fastly’s monitoring spotted its outage within a minute. The team disabled the setting that triggered the bug and had 95% of its network back within 49 minutes. The fix for the bug itself rolled out later that day. Detect, mitigate, restore, then fix and write it up.