Atomic state machine

The atomic-updates pattern starts from a race. A read-modify-write — read a value, compute a new one, write it back — is three steps, and any other client can act in the gaps. The source's worked example is a balance: read account:123:balance, subtract 100, write the result. Two clients that both read the old balance before either writes both compute from the same number, and one update is lost.

Applied to the queue, the value being updated is not a balance but a job's location. A worker finishing a job reads the lock on the job in active, branches to completed, and writes the move. A recovery sweep, running at the same instant, reads the same job still in active and reclaims it to wait. Run as separate commands, the two interleave: the job lands in completed and wait both, or its lock is cleared by one process while the other still treats it as held. The concurrent writers are the finishing worker and the recovery sweep (or a second worker), and the value they race on is the job's place in the queue.

The figure below runs the finish over a fixed set of jobs. Pick a transition and watch the job id move from one location to the next; with fetch_next on, the same call also pulls the next waiting job into active. The readout names the move, the structures it touches, and that it is one EVALSHA.

A Lua script executes atomically — no other command runs between the script's first line and its last. So the fix is to fold the whole read-decide-write into one script and run it with a single call. The read (is the lock still held? is this the right token?), the decision (completed or failed; past max attempts?), and every write (clear the lock, record the result, move the id, bump metrics, fetch the next) happen as one indivisible step. No second client can interleave, because Redis runs the script to completion before it serves the next command.

In EchoMQ this script is moveToFinished-14.lua, dispatched by EchoMQ.Scripts.move_to_finished/7. It touches fourteen keys — the whole queue shape — and it does the finish AND the next fetch in one round trip: fetch_next defaults to 1, so the same call that finishes the current job pops the next one from wait into active. The lifecycle transition and the next fetch are one atomic move.

# moveToFinished-14.lua — one EVALSHA, the whole lifecycle transition (real, Elixir)
# KEYS  1 wait      2 active   3 prioritized  4 event-stream  5 stalled
# KEYS  6 limiter   7 delayed  8 paused       9 meta         10 pc
# KEYS 11 completed|failed  12 jobId  13 metrics  14 marker
EVALSHA <sha> 14 wait active prioritized events stalled limiter \
        delayed paused meta pc completed jobId metrics marker \
        jobId ts returnvalue result completed 1 ...     # ARGV[6] fetch_next = 1
# clear the lock · write the result on the job hash · move jobId active → completed
# bump metrics · pop the next job wait → active — all in one indivisible step

The client issues exactly one command per transition: the EVALSHA. The script body runs the queue moves — RPOPLPUSH over the wait/ active lists, ZADD over the completed/failed sorted sets, HSET on the job hash — but those reach Redis inside the one cached-script call.

Use the atomic-script transition whenever a job's move reads state, branches on it, and writes across more than one key — and a second process can run the same move at the same time. A queue's finish is exactly that: it reads the lock and the attempt count, branches to completed or failed, and writes across the lists, the sorted sets, the job hash, and the metrics. A recovery sweep runs concurrently by design. The transition must be one step.

A single-key, single-command update needs no script — INCR, SET … GET, and GETDEL are already atomic on their own. Reach for the script only when the move spans multiple keys or carries a branch. And keep the keys on one node: a Lua script may only touch keys in the same slot, so a clustered queue co-locates its keys with a hash tag (the R2.05 treatment) before a fourteen-key script can run at all.

Each dive takes one part of the transition, in order: a job's states are Redis locations, the move between them is one EVALSHA over fourteen keys, and the cached script reaches the server by SHA with a NOSCRIPT fallback. Read them in order — the locations first, the atomic move second, the SHA mechanics third.

The scripting commands behind the transition — the cached call, the load, the full-body fallback — and where the engine that runs them is taught.