Meta / Facebook · 2021 · ~6 hours
A routine maintenance command took the whole platform off the internet
A maintenance command intended to check backbone network capacity instead withdrew the BGP routes that told the internet how to find Facebook's own DNS servers. Once those routes vanished, nothing — including Facebook's own engineers — could find Facebook.
Systems Involved
Timeline Replay
T+0m
Routine command audits backbone router capacity across regions.
T+1m
A bug in the audit tool withdraws BGP routes to Facebook's DNS servers.
T+2m
Facebook's DNS becomes unreachable from the rest of the internet.
T+5m
Internal tools that depend on the same DNS also go dark — including badge access to data centers.
T+45m
Engineers are physically dispatched to data centers to regain access.
T+5h50m
BGP routes are safely re-announced and DNS resolution recovers.
Root Cause
An automated audit tool had a bug that, instead of just checking backbone capacity, issued a command that withdrew BGP routes advertising Facebook's DNS servers to the rest of the internet. With no route to the DNS servers, no other system — internal or external — could resolve any Facebook domain.
Recovery
Because the outage also disabled the internal systems used to manage the network remotely, engineers had to physically go to data centers to reset hardware and safely restore the BGP announcements — a slow, manual process by design, since network changes at that scale are usually made very deliberately.
Lessons Learned
- Your break-glass emergency access shouldn't depend on the same systems it might need to fix.
- Automated network tooling needs the same blast-radius guardrails as automated deploys.
- DNS is a single point of failure for almost everything, even for the company that runs it.