The reliable queue

RPOP and process is the leak. The job leaves the queue the moment it is popped, before any work has happened; if the worker crashes mid-job — a deploy, an OOM kill, a power loss — the job is neither in Redis nor finished. It is lost, with nothing to recover it from.

The fix is to never let a job leave Redis until it is done. Instead of popping the job out, move it into a second list — an in-flight list — in one atomic step. RPOPLPUSH source dest (and its modern successor LMOVE source dest RIGHT LEFT) pops from one list and pushes to another as a single command: the job is in exactly one of the two lists at every instant, never in neither. A job parked in the in-flight list whose worker has died is still there — it can be found and returned for redelivery.

# the leak: a window where the job is in neither list
job = RPOP queue            # job is now out of Redis...
process(job)                # ...if the worker dies HERE, the job is gone

# the fix: the job is always in exactly one list
job = RPOPLPUSH queue processing   # atomic: leaves `queue`, lands in `processing`
process(job)                       # a crash here leaves the job in `processing`
LREM processing 1 job              # remove only after the work is truly done

RPOPLPUSH is the original move command; LMOVE (Redis 6.2+) generalises it with explicit directions and is the recommended form for new code. Either way the guarantee is the same: there is no instant at which the job exists nowhere.

The in-flight move makes redelivery possible — that is the whole point. But it also makes redelivery inevitable in one case: a worker that finishes the work, then crashes before it can remove the job from the in-flight list. On recovery the job is still in-flight, so it is redelivered and the work runs again.

This is the honest cost of never losing a job. Exactly-once delivery is a lie — you cannot both guarantee a job is never lost and guarantee it is never delivered twice, because the acknowledgement itself can be lost. What a reliable queue offers is at-least-once delivery: a job is delivered one or more times, never zero. The responsibility moves to the consumer, which must be idempotent — running the same job twice must produce the same effect as running it once.

The discipline is to make the effect exactly-once even though the delivery is at-least-once. The standard move is a dedup marker keyed by a stable job id: before the effect runs, claim the marker; if it already exists, the effect has already happened, so skip it.

# at-least-once delivery + an idempotent effect = exactly-once effect
if SET dedup:{job_id} 1 NX:          # claim the marker; NX = only if absent
    apply_effect(job)                # first delivery: charge the card once
else:
    skip()                           # a redelivery: marker is set, do nothing

A non-idempotent consumer double-charges on every redelivery; an idempotent one charges once, no matter how many times the job arrives. The marker is the difference: the second delivery finds it set and skips the effect. Toggle the marker off below and slide the delivery count up to watch the effects multiply.

A job in the in-flight list whose worker has died must come back. The signal that a worker died is a lock that expired: each in-flight job holds a lock with a TTL, and a healthy worker renews it on a heartbeat. When the worker crashes, the heartbeat stops, the lock expires, and the job is now stuck in-flight with no live worker.

A separate recovery sweep finds these jobs and returns them to wait for another worker. It checks the in-flight list, and for each job whose lock has expired, moves it back to the waiting list. The job re-enters the normal pickup path and is delivered again — at-least-once in action — to a worker that can finish it. A job is only truly lost if both the recovery sweep and every worker fail at once, which is the failure the lock TTL plus the sweep are designed to bound.

The recovery sweep must itself be careful. If it runs the check and the move as two separate steps, two recovery processes can both see the same stalled job and both move it, redelivering it twice over. The atomic form runs the detect-and-move as one step — one Lua script over the in-flight list, the lock, and the waiting list — so each stalled job is reclaimed once per sweep. The non-atomic form, where the check and the move are separate round trips, is the cautionary contrast: correct only until two sweeps overlap.

EchoMQ's worker fetch loop is all three guarantees in real code, in echo/apps/echomq. The in-flight move is moveToActive-11.lua: its body runs rcall("RPOPLPUSH", waitKey, activeKey), where waitKey is emq:{queue}:wait and activeKey is emq:{queue}:active (EchoMQ.Keys.wait/1 and EchoMQ.Keys.active/1). The job leaves wait and lands in active under a lock in one atomic script — a crash leaves it in active, recoverable, never lost.

-- moveToActive-11.lua — the atomic in-flight move (real)
local waitKey = KEYS[1]      -- emq:{queue}:wait
local activeKey = KEYS[2]    -- emq:{queue}:active
local jobId = rcall("RPOPLPUSH", waitKey, activeKey)  -- wait -> active, atomically

At-least-once with an idempotent effect is the dedup marker EchoMQ.Keys.dedup/2, which builds emq:{queue}:de:{id} ("#{base(ctx)}:de:#{dedup_id}"). A job carrying a deduplication id is dropped if the marker already exists, so the effect runs once across redeliveries. Stalled recovery is moveStalledJobsToWait-8.lua: it operates over the stalled, wait, and active keys, and for each job in active whose lock has expired it moves the job back to wait for another worker. The Go port's non-atomic stalled path — a separate check followed by a separate move — is the cautionary contrast to this atomic form, not the recommended one.

Where the atomic move that parks a job in the in-flight list is specified, its modern successor, the scripting that runs the multi-key reliable move as one step, and the queue protocol EchoMQ ports.