Platform Engineering Meetup

p99 under 200 ms

What we learned replacing a polling job queue with partitioned leases.

Dana Ortiz, Staff EngineerFictional case studyOctober 2026

Our slowest 1% of requests took 2.4 s.

p99 latency, enqueue to ack

2.4 s

Median was 90 ms. The tail was twenty-six times worse.

Before

Workers polled a Postgres table

Simplified architecture of the old queue.

The fix

2

Lease, don’t scan

Partition the work, then claim it without blocking.

After

Partitioned lanes, one lease each

Each worker owns a lane; claims skip rows another worker holds.

Life of a job

Four steps, no polling

Enqueue
The API writes the job to its lane, chosen by hashing the tenant.
Lease
A worker claims a batch for 30 seconds.
Execute
Work runs; the lease is renewed while it does.
Ack
The row is deleted. A missed ack simply expires.

Lease

The whole trick is one query

    -- claim up to 10 ready jobs without waiting on anyone
    UPDATE jobs SET state = 'leased',
      lease_until = now() + interval '30 seconds'
    WHERE id IN (
      SELECT id FROM jobs WHERE state = 'ready' AND lane = 2
      ORDER BY run_at LIMIT 10 FOR UPDATE SKIP LOCKED)
    RETURNING *;

Tail latency fell over eight weeks of rollout

Week	p99
W1	2400
W2	2350
W3	1500
W4	820
W5	410
W6	260
W7	195
W8	180

p99 enqueue-to-ack latency in milliseconds, weekly. Sample data.

What we considered

	Keep polling	Redis list	Partitioned leases
p99 at load	2.4 s	~300 ms	180 ms
New infrastructure	None	Redis cluster	None
If a worker dies	Job stuck	Job lost	Lease expires

Looking back

What worked

Hashing by tenant kept lanes even
Shadow-writing to both queues for two weeks
One dashboard: lease age, not queue depth

What we’d change

Start with 16 lanes, not 4
Alert on lease expiry rate from day one
Write the rollback runbook before the migration

Result after 90 days

13×Lower p99 latency

62%Less database CPU

0Jobs lost

Thanks for listening

Questions?

Slides example.dev/p99Dana dana@example.dev