Redis Cluster Failure: Incident Report

On March 3, 2025, at 11:42 UTC, our production Redis cluster experienced a primary replica failure that led to a 2.5-hour period of degraded performance. The primary node crashed due to severe memory fragmentation following a large key eviction event. Failover to the replica was delayed because the replica was still syncing a large RDB snapshot. During the failover window, all cached sessions were invalidated, forcing ~180,000 users to re-authenticate. After failover completed, a cache stampede drove unprecedented load to the primary PostgreSQL database. Root cause was insufficient Redis memory headroom and a cache key design that allowed unbounded growth.

The primary root cause was memory fragmentation in the Redis primary. A poorly designed cache key pattern (user activity feed) had grown to ~40M keys without TTL. When memory pressure triggered eviction, Redis attempted to evict large hash objects. The eviction process itself caused additional fragmentation, and the allocator could not coalesce freed memory. The process hit the configured maxmemory limit and was killed by the OOM killer.

A contributing factor was the failover delay. The replica was 8 minutes behind the primary due to a large RDB snapshot transfer. During this window, Redis was unavailable for writes, and all in-memory session data was lost. We had not configured Redis to use AOF (append-only file) for session data.

The cache stampede was a secondary effect. When Redis came back online, every request was a cache miss. Thousands of requests simultaneously queried PostgreSQL for session and user data, exhausting the connection pool.

• Duration: 2.5 hours (degraded), 13 minutes (full outage)
• Sessions lost: ~180,000 users logged out
• Database impact: Connection pool exhaustion, P99 latency > 5s for 1 hour
• Revenue impact: Checkout abandonment rate 3x normal during window
• Customer support: 2,100 tickets related to "logged out" or "session expired"

Detection (11:38–11:42 UTC): Redis memory crossed 90% at 11:38. Primary OOM crash at 11:42. Alert fired within 30 seconds. MTTD: ~4 minutes from first warning to full outage.

Response (11:42–11:52 UTC): Platform team engaged. Replica promoted but 8 minutes behind. Manual failover decision. Session cache invalidated: all users logged out. Auth service overwhelmed by DB fallback.

Resolution (11:52–14:12 UTC): Replica finished sync at 11:52. Cache stampede began: DB connection pool exhausted. Read replicas added at 12:15. Memory fragmentation addressed by 12:48. All caches warm by 14:12. MTTR: 2.5 hours.

1. Why did users get logged out? → Redis primary crashed, all session data lost.
2. Why did Redis crash? → OOM killer terminated process due to memory exhaustion.
3. Why was memory exhausted? → Unbounded cache key growth (40M keys, no TTL) + eviction caused fragmentation.
4. Why was cache unbounded? → Activity feed cache design had no TTL or size limit.
5. Why wasn't this caught earlier? → ROOT CAUSE: No memory headroom monitoring, no cache key audits, replica lag not alerted.

• Add TTL to all Redis cache keys (max 24h for session, 1h for feed)
• Enable AOF for session persistence
• 30% memory headroom for Redis
• Cache stampede protection (request coalescing, probabilistic early expiration)
• Alert on replica lag > 60 seconds
• Regular cache key audits and size limits

[ERROR] Redis primary OOM killed. Replica lag: 8m 12s. Promoting replica...

[WARN] auth-service Session validation falling back to DB - cache miss rate: 94%

[ERROR] PostgreSQL connection pool exhausted (200/200). P99 latency: 5200ms

• Unbounded cache growth is a ticking time bomb. The activity feed cache had no TTL and no size limit.
• Memory fragmentation can cause sudden OOM. Regular memory profiling and headroom are essential.
• Failover is not instant. Replica sync lag meant 13 minutes of downtime. AOF would have reduced data loss.
• Cache stampede is a real risk. We need stampede protection for critical caches.
• Redis cluster failure root cause analysis: this postmortem documents our OOM and failover recovery.