Read-decide-write in one EVALSHA

Finishing a job is not one write. It is a read, a decision, and several writes:

• Read — is the lock still held, and is this the right token? A worker that lost its lease must not finish a job a recovery sweep has already reclaimed.
• Decide — completed or failed; and past the attempt ceiling, failed instead of retried.
• Write — clear the lock, record the result on the job hash, move the id from active to completed or failed, bump the metrics, and fetch the next job.

That is a read-modify-write across many keys. Done as separate commands, the gaps between them are open. A recovery sweep reads the same job still in active, reclaims it, and writes it back to wait — between the finishing worker's read and its write. The job lands in both completed and wait, or its lock is cleared by one process while the other still treats it as held. At-least-once becomes a mess by accident.

A Lua script runs to completion before Redis serves the next command. So folding the whole read-decide-write into one script and dispatching it with a single EVALSHA removes every gap. The read, the decision, and the writes are one indivisible step; no recovery sweep and no second finish can interleave.

-- moveToFinished-14.lua — the read-decide-write as one EVALSHA (real, Elixir)
-- read:   is the lock held under this token? what is the attempt count?
-- decide: target is `completed` or `failed`; past max attempts → failed
-- write:  clear the lock · record the result on the job hash
--         move jobId active → completed|failed · bump metrics
--         (fetch_next=1) pop the next job wait → active
-- nothing runs between the first line and the last — one indivisible step

The script is the lock. Because the read, the decision, and the writes are one step, there is no window between them for a sweep to act in. EchoMQ.Scripts.move_to_finished/7 dispatches it with execute(conn, :move_to_finished, keys, args), a named-script call that runs EVALSHA.

moveToFinished-14.lua touches the whole queue shape — fourteen keys, named in this exact order. EchoMQ.Scripts.move_to_finished/7 builds the list in this order and passes it to the one EVALSHA:

1. 1 emq:{queue}:wait LIST
2. 2 emq:{queue}:active LIST
3. 3 emq:{queue}:prioritized ZSET
4. 4 emq:{queue}:events STREAM
5. 5 emq:{queue}:stalled SET
6. 6 emq:{queue}:limiter
7. 7 emq:{queue}:delayed ZSET
8. 8 emq:{queue}:paused LIST
9. 9 emq:{queue}:meta
10. 10 emq:{queue}:pc priority counter
11. 11 emq:{queue}:completed | :failed ZSET · the target
12. 12 emq:{queue}:<id> job hash
13. 13 emq:{queue}:metrics:<target>
14. 14 emq:{queue}:marker

All fourteen reach Redis as one command. The completed-or-failed key in slot 11 is the target the target argument selects (:completed or :failed); the wrappers move_to_completed/6 and move_to_failed/6 both delegate to move_to_finished/7 with that target set.

The centrepiece is one ARGV flag. fetch_next defaults to 1, so the same moveToFinished call that finishes the current job also pops the next id from wait into active — under the same atomic step. Finish-then-fetch, the two moves a worker makes back to back, is one round trip rather than two. The lifecycle transition and the next fetch are indivisible: there is no instant between the finish and the next pickup at which the queue sits half-moved.

A worker loop that finished a job and then made a second call to fetch the next would pay two round trips and open a window between them. EchoMQ's loop pays one. The slider below contrasts the round-trip count directly.

The cached-script call that runs the transition as one step, why a script is indivisible, the target structures the move writes into, and the queue protocol EchoMQ ports.