The lock protocol

While a worker runs a job, the job is owned by one key — a UUID token under a TTL — and kept alive by a heartbeat.

The lock is a single Redis key, EchoMQ.Keys.lock/2, holding a UUID v4 token under a PX TTL. Pickup acquires it inside moveToActive-11 with SET …:lock <uuid> PX lockDuration; a heartbeat keeps it — extendLock-2.lua re-sets the TTL every lockDuration/2 after verifying the token; completion releases it with DEL; and a missing or mismatched token returns -2 or -6. Move the renew interval on the right to see whether a job stays owned or stalls.

Acquire, heartbeat, verify The token check The timing The Golden Rule References

The lock timeline · set the renew interval

renew interval 15000 ms

Acquire, heartbeat, verify

The lock key is EchoMQ.Keys.lock/2 — "#{job(ctx, job_id)}:lock", that is emq:{queue}:{jobId}:lock. It has three moves.

Acquire is part of pickup: inside moveToActive-11, the script sets SET emq:{queue}:{jobId}:lock <uuid-token> PX <lockDuration>. The token is a UUID v4 unique to the worker; the PX TTL auto-expires the lock if the worker crashes, so a dead worker's job is not owned forever.

The heartbeat is extendLock-2.lua — two keys ([1] the lock, [2] the stalled set), reached from Elixir as EchoMQ.Scripts.extend_lock/5. Its whole body is twenty-three lines:

-- extendLock-2.lua · KEYS[1] lock, KEYS[2] stalled · ARGV token, duration-ms, jobid
local rcall = redis.call
if rcall("GET", KEYS[1]) == ARGV[1] then
  if rcall("SET", KEYS[1], ARGV[1], "PX", ARGV[2]) then
    rcall("SREM", KEYS[2], ARGV[3])
    return 1
  end
end
return 0

It verifies the token (no one else holds the lock), re-sets the TTL, and removes the job from the stalled set — returning 1 on success and 0 if the token no longer matches. A worker renews every lockDuration/2.

Verification is the other transitions' lock check. moveToFinished-14 confirms the token before completing; a missing lock returns -2 (JobLockNotExist), a mismatched token -6 (JobLockMismatch). Release on success is the DEL step inside moveToFinished-14.

The token check

The heartbeat's first line is the guard: GET lock == ARGV[1] token. Pick the two outcomes below to read what the script does in each.

extendLock-2 · the token check · pick an outcome

The heartbeat changes nothing unless the token still matches. A renewal that returns 0 means the lock is no longer the worker's — taken by another, or already lapsed.

The timing

Four parameters set the rhythm, with these defaults (ch03 §3.4).

lockDuration: 30000 ms — the lock TTL set by SET … PX.
lockRenewTime: 15000 ms (= lockDuration/2) — the heartbeat interval.
stalledInterval: 30000 ms — the stalled-check period.
maxStalledCount: 1 — max stalls before permanent failure.

A worker renews every lockDuration/2, so a healthy job's lock never expires: each renewal extends the 30000 ms TTL while only 15000 ms has elapsed. If a worker crashes and misses a renewal, the TTL runs out; the next stalled sweep (moveStalledJobsToWait-8) finds the job in active with no lock and moves it back to wait — or to failed once stc reaches maxStalledCount. The hero timeline is exactly this arithmetic: a renew interval at or under lockDuration keeps the job owned; longer lets the lock lapse and the job stall.

The heartbeat above renews one lock. In production, a worker holds many jobs and renews them together: EchoMQ's Elixir runtime drives one EchoMQ.LockManager timer per worker that batch-renews every tracked job via EchoMQ.Scripts.extend_locks/5 (extendLocks-1.lua, one stalled key, the job ids and tokens packed into ARGV) — the one-timer story, taught in the lock-management module of this chapter.

The Golden Rule — the lock protocol is L1/L2

The lock key, the SET …:lock <uuid> PX acquire, the extendLock-2.lua heartbeat, and the -2/-6 verification codes are L1/L2 — immutable and shared. The same lock protocol runs in every runtime; only the executor that issues the heartbeat and the renew schedule above it vary.

the protocol (immutable L1/L2)

EchoMQ.Keys.lock/2, the SET …:lock <uuid> PX lockDuration acquire, the extendLock-2.lua heartbeat, and the closed -2/-6 lock codes — the same in every runtime.

its three runtimes (variable L3/L4)

Elixir issues the heartbeat with EchoMQ.Scripts.extend_lock/5 over a Redix pool and schedules renewals with one EchoMQ.LockManager timer per worker; Go and Node.js run the same extendLock-2.lua their own way. The lock protocol does not move; the schedule above it does.

A job is owned by a UUID token under a TTL; the heartbeat re-sets it every lockDuration/2 after a token check, and a missing or mismatched token returns -2/-6. The lock protocol is L1/L2 — the same in every runtime.

This is the last dive in E2.01. The chapter continues with the components and the lock management; the workshop runs a job through every state — reachable from E2 · the core.

References

The Redis commands the lock protocol issues, and the wire protocol that fixes its timing.

Sources

BullMQ — Documentation — the lock protocol: acquire, heartbeat, verification, and the stalled-recovery timing.
Redis — SET — the SET key value PX ms acquire and TTL re-set.
Redis — SREM — the remove-from-stalled-set step the heartbeat performs.
Redis — GET — the token read the heartbeat verifies before re-setting the lock.

Related in this course

E2.01 · The lifecycle & state machine — the module hub.
E2.01.2 · Every transition & its script — where -2/-6 are emitted, and the pickup that acquires the lock.
E1 · The protocol — the key taxonomy the lock key belongs to.
redis-patterns R2 · Coordination — the lock-with-fence pattern, applied.