GitHub · 2018 · ~24 hours
A 43-second network partition led to over a day of database inconsistency
A brief network partition between two East Coast data centers caused GitHub's MySQL cluster to fail over in a way that left two data centers each believing they held the authoritative copy of the data — forcing a slow, careful reconciliation instead of a fast failover.
Systems Involved
Timeline Replay
T+0s
A network link between two data centers fails for 43 seconds.
T+43s
The failover orchestrator promotes a replica in a different data center to primary.
T+50s
The network link recovers, but two data centers now disagree on which node is primary.
T+2m
Writes have landed in both locations — the cluster is in a split-brain state.
T+20m
Site is put into a degraded, read-only-where-possible mode to stop further divergence.
T+24h
Data is manually reconciled and full read/write service is restored.
Root Cause
The automated failover correctly detected a problem and did what it was designed to do — but a same-second network blip meant the old primary hadn't fully stepped down before a new one was promoted elsewhere, and a short window of writes landed on both.
Recovery
Rather than risk losing data, the team chose consistency over speed: they degraded functionality to stop the divergence from growing, then spent hours carefully reconciling records that had been written to both locations before fully reopening writes.
Lessons Learned
- Automated failover needs to account for the old primary not going away cleanly, not just a clean handoff.
- Sometimes the correct incident response is to choose a longer outage over a faster one that risks data loss.
- A 43-second blip can matter more than a 43-minute one, depending on exactly what it touches.