Stalled recovery

The source pattern runs a separate monitor process — sometimes called a reaper — that periodically scans the processing queues for stalled messages. For each message the reaper applies a timeout rule: check each processing queue; if a message has been there longer than a timeout (for example, ten minutes), the worker is assumed dead; move the message back to the main queue with LMOVE processing:worker1 work_queue RIGHT RIGHT. This guarantees at-least-once delivery — a message may be processed more than once, but it is never lost.

EchoMQ keeps the shape of the reaper and sharpens its test. The reaper's test is has this message been in-flight longer than a fixed timeout? — a coarse proxy for a dead worker that misfires whenever a job legitimately runs long. EchoMQ's test is more precise: does this in-flight job still have a live worker? The answer is a lock lease. Each in-flight job holds a lock with a PX expiry, and a healthy worker renews that lease on a heartbeat. When the worker dies, the heartbeat stops, the lease expires, and the lock is gone. Lock gone means the worker died means the job stalled. The timeout becomes a lease, and the lease is exact: it is tied to the worker, not the clock.

The sweep below runs over a fixed in-flight set. Toggle a worker dead to drop its lease, then run the sweep: the jobs whose lock is gone move back to wait, the jobs whose worker is still renewing stay put. The readout reports which jobs are stalled and where each lands.

A lock that expired is the signal a worker died. The lease (EchoMQ.Keys.lock/2 → emq:{queue}:<id>:lock) is a key with a short PX TTL set when a worker takes a job. A live worker renews it on a heartbeat — SET … PX, a lease renewal — before it can lapse. The renewals run in batch from one timer in EchoMQ.LockManager, so a pool of workers extends a pool of leases without one timer per job.

# the lease: set when a worker takes the job
SET emq:{queue}:<id>:lock <token> PX 30000     # a 30s lease

# the heartbeat: a live worker renews before the lease lapses
SET emq:{queue}:<id>:lock <token> PX 30000     # pushes the expiry forward, every ~half-lease

# the death signal: no renewal -> the TTL runs to 0 -> Redis evicts the key
EXISTS emq:{queue}:<id>:lock                  # 0 means the worker died -> stalled

When the worker crashes, the heartbeat stops. No renewal arrives, the TTL runs to zero, and Redis evicts the lock key. The job is now in active with no live worker. The recovery sweep tests exactly this — for each job in active, is the lock key still present? — and a missing lock is a stalled job. The first dive builds this heartbeat-and-lease picture, and shows why a paused worker that keeps renewing is correctly left alone while a dead one is reclaimed.

A brief pause is not a dead worker. A worker that misses one heartbeat — a long GC pause, a slow disk — may renew on the next tick and finish the job. If the sweep reclaimed a job the instant its lock lapsed, every momentary stall would be redelivered, doubling the work.

The fix is a two-phase sweep. The first sweep that finds a job stalled marks it: it records the job in a stalled set rather than reclaiming it on the spot. Only the second sweep that still finds the job stalled recovers it. A momentary stall survives one grace pass; a genuinely dead worker is reclaimed on the next sweep. When a job is recovered, its attempt count is incremented and it goes back to wait — unless it has now exceeded its maximum attempts, in which case it goes to failed instead of being retried forever. The second dive walks the mark → recover → wait-or-failed decision.

The recovery sweep must itself be careful. The detect step tests is this job stalled? and the recover step does remove it from active, push it to wait. If those are two separate round trips, two sweeps running at once can both pass the detect step for the same job and both run the recover step — the job lands in wait twice, and its attempt counter is mis-incremented. This is the double-recovery window.

The atomic form closes it. moveStalledJobsToWait-8.lua runs the whole detect-and-move as one EVALSHA: the read, the decision, and the writes happen inside Redis's single thread, so no second sweep can interleave. Each stalled job is reclaimed once per sweep, and the attempt count is incremented in the script. The Go port's StalledChecker does the same work app-side — LRANGE active, then per job EXISTS lock, then LREM/LPUSH — across separate round trips, and increments the attempt count client-side. It is correct until two checkers overlap; then the window opens. The third dive is the centrepiece: it runs the two overlapping checkers and counts the recoveries.

A recovery sweep belongs to any reliable queue whose workers can crash mid-job. The choices it makes are how it detects a dead worker, whether it grants a grace pass, and how the detect-and-move is joined.

Each dive takes one part of the sweep, in order: how it detects a dead worker, how it marks before it recovers, and why the detect-and-move must be one indivisible step. Read them in order — the lease comes first, the two-phase decision second, the atomic contrast third.

The Redis mechanisms behind the recovery sweep — the lease, the atomic script, the removal, and the move — and where the engine that runs them is taught.