Blameless review

The cache stampede

How one expired key took the profile service down for eleven minutes, and the 14 lines that fixed it.

Platform teamFictional incidentOctober 2026

One key expired. 4,000 requests asked the database the same question.

Timeline, UTC

Eleven minutes

  1. Key expires

    Popular profile, TTL 300 s.

  2. Pool exhausted

    Every miss opens a connection.

  3. Page fires

    p99 above 8 s.

  4. Recovered

    Traffic shed, cache warm.

The fix

One fetch per key, everyone else waits

    async function getProfile(id) {
      const hit = await cache.get(id)
      if (hit) return hit
      // single flight: share one in-flight fetch per key
      if (!inflight.has(id)) {
        inflight.set(id, db.load(id).finally(() => inflight.delete(id)))
      }
      return inflight.get(id)
    }

p99 latency after the fix

Same load test, same key expiry.

Buildp99
Before8200
After180
Replay of the incident traffic, staging

What worked, what did not

Worked

  • Runbook found the pool graph in two minutes
  • Load shedding bought time
  • Replay harness reproduced it first try

Did not

  • Page threshold was 8 s, not 1 s
  • No jitter on TTLs
  • Dashboards showed averages only

Action items