Illustrative: Payments Platform · Illustrative · ~50 minutes
A schema migration silently exhausted the database connection pool
An illustrative, generalized scenario modeled on a common failure pattern at payment companies: a routine schema migration held a lock longer than expected, backing up connections until the pool was exhausted and new payment requests started failing.
Systems Involved
Timeline Replay
T+0m
A routine migration begins, adding an index to a high-traffic table.
T+4m
The index build takes a lock that ordinary queries also need.
T+8m
Queries start queuing behind the lock; the connection pool fills up.
T+11m
New payment requests start timing out waiting for a free connection.
T+30m
On-call cancels the migration and the lock is released.
T+50m
The connection pool drains and payment success rate returns to normal.
Root Cause
The migration was tested against a much smaller dataset than production, where the same index build completed in milliseconds. At full production scale, the lock was held long enough to back up every other query behind it.
Recovery
On-call killed the migration to release the lock immediately, then re-ran it later using an online schema change tool that avoids long-held locks on large tables.
Lessons Learned
- Test migrations against production-scale data, or at least production-scale locking behavior.
- Connection pool exhaustion is often a symptom — the real cause is usually a lock, a leak, or a slow dependency.
- Prefer migration tools designed for zero-downtime schema changes on hot tables.