Kubernetes Crash Loop: Incident Report

On February 7, 2024, the checkout-api service entered a crash loop shortly after deploying v2.15.0. Pods repeatedly crashed with panic: nil pointer dereference in PaymentService.Process(). The incident lasted 18 minutes and affected approximately 8,000 users attempting checkout. Root cause was a missing nil check in the new payment flow code. Rollback to v2.14.3 restored service within 6 minutes of the decision.

The primary root cause was a nil pointer dereference in PaymentService.Process() introduced in checkout-api v2.15.0. The new code path did not validate that the Stripe client was initialized before use when Redis session cache was unavailable. Under the cascade (Redis failover + Stripe latency), the code hit the unvalidated path and panicked.

A contributing factor was the Redis cluster failure, which increased auth-service load and caused 94% cache miss rate. This shifted load to the database and created additional latency that exposed the nil pointer path more frequently.

• Duration: 18 minutes (checkout unavailable)
• Users affected: ~8,000 checkout attempts
• Support tickets: 340 in 5 minutes
• Revenue impact: Checkout abandonment during window
• Root cause: Nil pointer in checkout-api v2.15.0 PaymentService.Process()

Detection (09:00:00–09:00:02 UTC): The incident was detected within 2 seconds when the first checkout-api pod panicked. PagerDuty INC-8842 was auto-created via our Slack integration. Mean Time to Detect (MTTD): 2 seconds.

Response (09:00:02–09:00:35 UTC): On-call engineer @alice_sre acknowledged within 3 seconds. War room identified Stripe timeouts and Redis cluster failure as contributing factors. Decision to rollback was made 30 seconds after confirming the crash loop. Rollback initiated at 09:00:35.

Resolution (09:00:35–09:01:15 UTC): Rollback to v2.14.3 completed. Redis failover finished at 09:00:48. First successful checkout at 09:00:59. All pods healthy by 09:01:15. Mean Time to Resolve (MTTR): 59 seconds.

1. Why did checkout fail for users? → checkout-api pods were crashing in a loop, returning 502 errors.
2. Why were pods crashing? → A nil pointer dereference panic in PaymentService.Process() caused immediate process termination.
3. Why was there a nil pointer dereference? → Code in v2.15.0 accessed a pointer without validating it was initialized.
4. Why was unvalidated code deployed to production? → The PR was approved without sufficient review, and no automated tests caught the nil case.
5. Why didn't tests or review catch this? → ROOT CAUSE: No mandatory nil-safety linting rules, insufficient test coverage for payment-critical paths, and no staged rollout (canary) to catch runtime errors before full deployment.

• Add nilaway or staticcheck linting to CI for nil-safety enforcement
• Require 90%+ test coverage for payment-critical code paths
• Mandatory canary deployment (5% traffic, 10-min bake) before full rollout
• Automated rollback trigger when error rate exceeds 20% for 30 seconds
• Circuit breaker tuning: fail-fast on upstream timeouts
• Redis cluster: automatic failover, 3-AZ spread

2024-02-07T09:00:00.123Z [FATAL] checkout-api-pod-7x9m2 panic: runtime error: invalid memory address or nil pointer dereference
goroutine 42 [running]: github.com/shop/checkout.(*PaymentService).Process(...)

{"ts":1707318005,"level":"error","msg":"Stripe API timeout","service":"payment-gateway","duration_ms":30000,"error":"context deadline exceeded"}

Feb 07, 2024 09:00:07 UTC [ERROR] auth-service Redis cluster: MOVED 15389 - connection to node failed after 3 attempts. Falling back to DB, cache miss rate: 94%

• Nil checks are critical in payment-critical paths. Defensive validation must be mandatory in code review.
• Canary deployments would have caught this. A 10% canary would have limited blast radius.
• Redis and Stripe failures can cascade. We need better isolation and circuit breakers between dependencies.
• Kubernetes CrashLoopBackOff is a common symptom: this postmortem documents how we diagnosed and resolved a nil pointer in production.