jonnifyechomq
EchoMQ / E2 · Core / The lifecycle / Every transition & its script

E2.01.2 · The lifecycle · dive 2

Every transition
& its script

Every move between states is one atomic Lua script — and the number in its filename is the count of keys it touches in one step.

Four scripts perform the moves the running library makes: moveToActive-11 picks a job up and acquires the lock, moveToFinished-14 completes it, moveToDelayed-8 retries it, and moveStalledJobsToWait-8 recovers a stalled one. Each runs its whole step list in one server-side execution, so a worker never observes a half-moved job. When a script cannot complete, it returns a negative integer from a closed set of eleven. Pick a transition on the right to read its KEYS arity and atomic step list.

Why one atomic script The four transitions The closed return codes The Golden Rule References

A transition · pick one

wait active moveToActive-11 KEYS = 11 each move = one atomic Lua script · the filename number is its KEYS arity

Why one atomic script

A state transition is several Redis writes — dequeue here, set the lock, push there, publish an event. Run as separate client commands they could interleave: a worker could observe a job in active with no lock, or in two state sets at once, if a process paused between two writes. EchoMQ runs each transition as one Lua script Redis executes server-side in a single step, so the whole step list is atomic — every observer sees the job either fully before or fully after the move, never in between.

The four transitions

Verified against priv/scripts/ and the per-operation functions in scripts.ex, these are the four moves and the keys each touches.

# transition          script                 KEYS   reached from Elixir
  pickup              moveToActive-11        11     EchoMQ.Scripts.move_to_active/4
  completion          moveToFinished-14      14     EchoMQ.Scripts.move_to_finished/7
  retry               moveToDelayed-8        8      EchoMQ.Scripts.move_to_delayed/6
  stalled recovery    moveStalledJobsToWait-8  8      driven by the stalled-checker process

Pickup (moveToActive-11, 11 keys) — promote due delayed → wait, check the rate limiter and the global concurrency, dequeue (RPOP wait or ZPOPMIN prioritized), acquire the lock (SET …:lock <uuid> PX), LPUSH active, XADD the active event, return the job's hash fields.

Completion (moveToFinished-14, 14 keys) — verify the lock token first, LREM active, HSET returnvalue or failedReason + stacktrace, ZADD completed/failed (score = timestamp), DEL the lock, increment metrics, apply removeOnComplete/ removeOnFail, update a parent's deps, clean the dedup key, XTRIM events, optionally return the next job. It is the most complex script in the protocol — roughly 1100 lines with the includes resolved.

Retry (moveToDelayed-8, 8 keys) — DEL the lock, compute the backoff delay = base * 2^(attempt-1) (optional jitter), move active → delayed with the new run-at score, increment atm (attempts made), XADD the delayed event.

Stalled recovery (moveStalledJobsToWait-8, 8 keys) — for each job in active, check whether the lock exists; no lock ⇒ stalled; if stc >= maxStalledCount move to failed, else move back to wait and increment stc.

The closed return vocabulary

When a script cannot complete, it returns a negative integer from a closed set — eleven codes, named, never expanded. Two of them are the lock codes moveToFinished-14 emits on its verify step. Pick a code below to read its name.

The closed -1 … -11 return codes · pick one

-1JobNotExist -5ParentJobNotExist -9JobHasFailedChildren -2JobLockNotExist -6JobLockMismatch -10SchedulerJobIdCollision -3JobNotInState -7ParentJobCannotBeReplaced -11SchedulerJobSlotsBusy -4JobPendingChildren -8JobBelongsToJobScheduler closed set · eleven codes · never expanded dashed = the two lock codes (-2, -6)

The two dashed codes are the lock codes: moveToFinished-14 verifies the token before completing, and returns -2 (JobLockNotExist) when the lock is missing or -6 (JobLockMismatch) when the token does not match the holder. The full set is the scripts' shared return convention — the lock protocol dive reads where -2/-6 come from.

The Golden Rule — the script is L2

A transition's script and its return codes are L2 — immutable and shared across the three runtimes. moveToActive-11 is eleven keys in a fixed order, performing the same atomic steps, in every runtime; the -1…-11 codes mean the same thing everywhere. Only the executor that dispatches the script by SHA varies.

the protocol (immutable L2)

The transition scripts — moveToActive-11, moveToFinished-14, moveToDelayed-8, moveStalledJobsToWait-8 — and the closed -1…-11 codes; the same atomic steps and the same codes in every runtime.

its three runtimes (variable L3)

Elixir dispatches them by SHA via EchoMQ.Scripts over a Redix pool; Go and Node.js load and run the same script files their own way. The script does not move between runtimes; the executor that dispatches it does.

Every move is one atomic Lua script whose filename number is its KEYS arity, and a job is never half-moved. The scripts and the -1…-11 codes are L2 — the same in every runtime.

Next: the lock protocol — the SET …:lock acquire inside moveToActive, the extendLock-2 heartbeat, and the token check behind -2/-6.

References

The dispatch that runs each script atomically, and the Redis commands the transitions issue.

Sources

  • BullMQ — Documentation — the transition scripts and the lifecycle moves they perform.
  • Redis — EVALSHA — the load-once, run-by-SHA dispatch every transition script runs under, atomically.
  • Redis — LREM — the remove-from-active step in moveToFinished.
  • Redis — ZADD — the add-to-terminal-set step (completed/failed, scored by timestamp).

Related in this course