What famous outages teach about blast radius, rollouts and backups
Updated Nov 10, 2026 · 5 min read
Big outages rarely have exotic causes. A typo, a config change, a forgotten server, a backup nobody tested. What turns a small mistake into a famous one is how far it spreads and how long it takes to undo. The companies below published honest write-ups of their worst days, and together they make a short course in designing for failure.
Blameless postmortems
A postmortem is the write-up after an incident. It sets out what happened, the timeline, why it happened, and action items to make it less likely or less harmful next time. Blameless means it asks what let the mistake happen, not who made it. If one typo can take down a region, the problem is the system that allowed it. Blame teaches people to hide mistakes, and hidden mistakes come back.
Seven outages in one table
| Outage | What happened | The lesson |
|---|---|---|
| Knight Capital, 2012 | New code reached 7 of 8 servers. A reused flag woke old code on the 8th, and removing the new code from the other 7 spread the problem. About $460 million lost in roughly 45 minutes | Automated deploys, no reused flags, kill switches |
| GitLab, 2017 | A wipe meant for a replica ran on the primary. The regular dumps had been failing, so the restore came from a snapshot about 6 hours old | Test restores, not just backups |
| AWS S3, 2017 | A mistyped input to a playbook command removed far more servers than meant, forcing two core subsystems to restart | Tools that refuse dangerous results |
| Cloudflare, 2019 | A firewall rule with a backtracking regular expression went to every server at once and pinned CPUs for about 27 minutes | Stage config like code, and bound regex run time |
| Fastly, 2021 | One customer’s valid config change hit a latent bug and 85% of the network returned errors. 95% recovered within 49 minutes | Fast detection, and a way to turn the trigger off |
| Facebook, 2021 | A maintenance command cut the backbone, DNS withdrew its routes, and the usual ways in were down too. About six hours | A way in that doesn’t depend on production |
| CrowdStrike, 2024 | A content update that wasn’t staged crashed Windows machines, about 8.5 million by Microsoft’s estimate | Staged rollouts and validation for content too |
Limit the blast radius
Blast radius is how much breaks when one thing goes wrong. The cheapest way to shrink it is a staged rollout. A change goes to a canary, a small slice of servers, first. You watch errors, CPU and crashes, then widen in steps, and roll back automatically if anything jumps.
The hard part is applying it to everything. Cloudflare’s rule and CrowdStrike’s update were both config or content rather than code, the kind of change that often skips the careful path because it needs to ship fast. Both companies committed afterwards to staged rollouts for exactly that kind of change. Stages can be fast. Minutes per stage still catches a change that crashes every machine it touches.
Cells shrink blast radius in space instead of time. Split the system into copies that share nothing, each serving a slice of customers. A bad change or a poison request then takes down one cell, not everyone. AWS’s S3 write-up describes moving up plans to split part of S3 into smaller cells after its outage.
Make tools refuse dangerous results
The S3 engineer was authorized and following a playbook. The tool simply allowed a typo to remove too much. AWS changed it to remove capacity slowly and to refuse any change that would take a subsystem below its minimum. Facebook had an audit tool meant to block commands like the one that cut its backbone, and a bug let it through. Guard rails belong in the tools, and the tools need tests too.
Test the restore, not the backup
GitLab had backups configured. When it needed them, the regular database dumps weren’t there, because the job had been failing and its warning emails were being rejected. Recovery came from a snapshot made to copy data to staging. A scheduled restore test catches this on the first day it goes wrong. Restore a recent backup somewhere safe, check the data looks right, and alert when it doesn’t, or when there’s no new backup at all.
Keep a way back in
When Facebook’s backbone went down, the normal ways into its data centers went with it, and losing DNS broke many internal tools. Engineers had to go on site, into buildings designed to be hard to get into. If your recovery tools depend on the thing that’s broken, the outage locks you out. Keep an out-of-band path, with its own network, logins and tools, and practise using it. Facebook also noted that turning everything back on at once could cause new failures from the surge, which is why it brought services back gradually.
Make rollback safe, and keep a kill switch
Knight Capital’s new code reused a flag that once switched on old code nobody used any more. One server missed the new code, so the flag woke the old code there. Removing the new code from the other seven made all eight do the same. A rollback only helps if the old version understands everything the new one has started sending. Never give an old flag a new meaning, delete dead code, check every server is running the same version, and keep a kill switch that stops everything at once.
Mitigate first, fix later
During an incident, the first job is to stop the damage, not to understand it. Fastly’s monitoring spotted its outage within a minute. The team disabled the setting that triggered the bug and had 95% of its network back within 49 minutes. The fix for the bug itself rolled out later that day. Detect, mitigate, restore, then fix and write it up.