Lock-expiry detection

The source's reaper scans the processing queue and, for each message older than a timeout, treats the worker as dead and moves the message back. The timeout is a proxy. It misfires in both directions: a job that legitimately runs longer than the timeout is reclaimed while its worker is still working, and a worker that dies one second into a ten-minute timeout holds a job hostage for the rest of it.

The lease removes the proxy. When a worker takes a job, it sets a lock key with a short PX expiry. A live worker renews that lease on a heartbeat — a periodic SET … PX that pushes the expiry forward — well before it can lapse. The lock is present exactly while a worker is alive and renewing. The reclaim question becomes a key lookup: for each job in active, is its lock key still present? A present lock is a live worker; a missing lock is a stalled job.

# the lease: set when a worker takes the job
SET emq:{queue}:<id>:lock <token> PX 30000     # a 30s lease

# the heartbeat: a live worker renews before the lease lapses
SET emq:{queue}:<id>:lock <token> PX 30000     # pushes the expiry forward, every ~half-lease

# the death signal: no renewal -> the TTL runs to 0 -> Redis evicts the key
EXISTS emq:{queue}:<id>:lock                  # 0 means the worker died -> stalled

The lease length and the heartbeat interval are a pair. The heartbeat must run several times within one lease so a single missed tick does not lapse it, and the lease must be short enough that a dead worker is detected promptly. A lease of thirty seconds renewed every ten is the usual shape: two missed renewals still leave the lock alive, three lapse it.

The lease distinguishes a worker that paused from a worker that died — a distinction a timeout cannot make. A paused worker that is still running its event loop keeps renewing the lease, so its lock stays present and the sweep leaves its job alone, even if the job has run for an hour. A dead worker stops renewing, its lock lapses within one lease, and the sweep reclaims its job. The test is the lock, not the clock: a long-running job under a live heartbeat is safe; a short job under a dead worker is reclaimed.

This is why the lease is more precise than the timeout it replaces. The timeout's test is how long has this job been running?, and it reclaims on a number. The lease's test is is a worker still renewing this lock?, and it reclaims on the worker. A job that runs longer than any fixed timeout but holds a live heartbeat is exactly the case the timeout gets wrong and the lease gets right. Toggle the two columns below to compare a slow-but-alive worker against a fast-but-dead one under both rules.

In EchoMQ the lease is EchoMQ.Keys.lock/2, which builds emq:{queue}:<id>:lock for a job id ("#{job(ctx, job_id)}:lock"). It is set with a PX expiry when the worker takes the job and renewed on the worker's heartbeat. The renewals do not run one timer per job: EchoMQ.LockManager keeps a single timer that extends all tracked leases in one batch, so a pool of workers renews a pool of leases without a timer storm.

# EchoMQ.Keys.lock/2 builds the lease key per job id (real, Elixir)
def lock(ctx, job_id), do: "#{job(ctx, job_id)}:lock"   # emq:{queue}:<id>:lock

# EchoMQ.LockManager renews all tracked leases from ONE timer, in batch —
# not one timer per job

The recovery sweep — moveStalledJobsToWait-8.lua — tests the lock for each job in emq:{queue}:active; a job whose lock has lapsed is the one with no live worker. The lease is the input to the whole sweep: the next dive marks those jobs, and the third runs the move atomically.

Where the lock lease is set and expired, the lookup the sweep runs, and the queue protocol whose lock-renewal path EchoMQ ports.