AWS (us-east-1) · 2021 · ~5 hours
An automated scaling activity exhausted network capacity in the largest AWS region
An automated scaling action in AWS's largest region triggered unexpected behavior in an internal network that many other AWS services — and AWS's own internal monitoring — depend on, causing a cascading, hard-to-diagnose regional incident.
Systems Involved
Timeline Replay
T+0m
An automated scaling activity begins on internal network infrastructure.
T+10m
Unexpected network device behavior causes internal service-to-service communication to degrade.
T+25m
Multiple dependent services begin returning errors across the region.
T+40m
Engineers struggle to get visibility, since internal monitoring shares the same affected network.
T+3h
Root cause identified; capacity is manually rebalanced.
T+5h
Dependent services recover as network capacity stabilizes.
Root Cause
A capacity-related automated action in the internal network fabric had a side effect that degraded internal communication at a scale large enough to affect many downstream services simultaneously — including, in a painful twist, some of the tools engineers would normally use to diagnose exactly this kind of problem.
Recovery
The team throttled the automated process responsible, manually rebalanced network capacity, and worked through the dependency graph service by service as communication was restored.
Lessons Learned
- At sufficient scale, your monitoring system needs its own independent blast-radius boundary from what it monitors.
- Automated scaling actions need rate limits and circuit breakers just like application code does.
- A single region hosting a disproportionate share of critical shared infrastructure is a systemic risk.