Skip to content

PATTERN Cited by 1 source

Lease-based task recovery

Problem

In a distributed task queue, workers can terminate mid-task due to VM eviction, out-of-memory conditions, or deployment events. If a terminated task remains marked as "processing," it is held indefinitely — a stuck task that no other worker will claim.

Heartbeat-based solutions (where workers periodically signal liveness) require an external coordination service and introduce complexity in distinguishing slow workers from dead ones.

Solution

Record an expiring lease (lease_expires_at timestamp) on the task row at dequeue time. A periodic sweeper process re-enqueues any task whose lease_expires_at has passed:

-- At dequeue time:
UPDATE tasks
SET status = 'processing',
    lease_expires_at = now() + interval '5 minutes'
WHERE id = (
    SELECT id FROM tasks
    WHERE status = 'enqueued'
    ORDER BY priority DESC, created_at
    LIMIT 1
    FOR UPDATE SKIP LOCKED
);

-- Periodic sweeper:
UPDATE tasks
SET status = 'enqueued', lease_expires_at = NULL
WHERE status = 'processing'
  AND lease_expires_at < now();

Tasks held by terminated workers are recovered automatically within the sweeper interval, without requiring an external coordination service or heartbeat protocol.

Key properties

  • No external coordination — the lease is stored in the same database as the task, requiring no Redis, ZooKeeper, or etcd.
  • Bounded recovery time — worst-case recovery latency equals the sweeper interval (typically minutes).
  • Compatible with idempotent processing — the re-enqueued task may be attempted by a new worker; the processing must be idempotent or the attempt table must track duplicates.
  • Simpler than heartbeats — no need to distinguish slow-but-alive workers from dead ones at the coordinator level; the lease is a hard deadline.

Trade-offs

  • Lease duration is a tension: too short → false recoveries of slow-but-alive tasks; too long → extended stuck-task windows after crashes.
  • Workers processing variable-latency tasks (e.g., 2-page invoice vs 200-page contract) need lease durations tuned to worst-case, or must implement lease renewal.
  • The sweeper itself is a single point of coordination that must run reliably.

Seen in

Last updated · 591 distilled / 1,796 read