Illustrative: Real-time Chat Platform · Illustrative · ~90 minutes
A cache stampede after a cold restart overwhelmed the primary datastore
An illustrative, generalized scenario modeled on a common failure pattern at real-time messaging platforms: a full cache-layer restart during a routine deploy caused every request to miss cache at once, sending the entire read load directly to a datastore sized for cache hit rates, not full traffic.
Systems Involved
Timeline Replay
T+0m
A deploy restarts the entire cache layer at once instead of node-by-node.
T+1m
Cache hit rate drops from ~95% to 0% instantly.
T+3m
The primary datastore receives the full, uncached read load and begins queuing.
T+6m
Read latency spikes across the platform; some messages fail to send.
T+40m
Traffic is throttled at the gateway to let the cache repopulate gradually.
T+90m
Cache hit rate recovers and normal latency returns.
Root Cause
The cache layer was restarted as a single unit rather than in a rolling fashion, so the system went from a warm cache to a completely cold one instantly. The datastore behind it was provisioned assuming the cache would always absorb the vast majority of reads.
Recovery
The team throttled incoming traffic at the gateway to reduce load on the datastore, allowing the cache to gradually repopulate under lighter pressure instead of being hit at full volume while empty.
Lessons Learned
- Never restart a cache layer as a single unit — roll it node by node, or warm it before cutting traffic over.
- If your datastore can't survive a 0% cache hit rate, your cache is load-bearing infrastructure, not an optimization.
- Gateway-level throttling is a useful tool for buying a struggling downstream system time to recover.