jonnifyechomq

E2.03 · Movement I · The Core

Lock management &
the supporting processes

A job is owned while a worker runs it, and three library processes keep that ownership honest.

A job in EchoMQ is owned by a lock — a UUID token under a PX TTL at emq:{queue}:{jobId}:lock. Three processes keep that ownership honest, and the host application supervises all of them, because EchoMQ-Elixir is library-only: EchoMQ.LockManager renews every held lock with one timer per worker, batched through extendLocks-1.lua; EchoMQ.StalledChecker recovers jobs whose worker died, in a two-phase mark-then-recover sweep over moveStalledJobsToWait-8.lua; and EchoMQ.JobScheduler plus EchoMQ.QueueEvents emit recurring jobs and consume the emq:{queue}:events Redis Stream. The lock key, the scripts, and the stream are L1/L2 — frozen and shared across the three runtimes. The core is shipped, so every surface below is read from the real source. Three dives follow.

Movement I · Core (as-built) ← redis-patterns R3

The renewal cost The dives How it fits References

One timer, or N — the renewal cost

The module's central choice is visible as arithmetic. A worker holds some number of jobs and must keep every held lock fresh. With a timer per job, N jobs cost N timers and N Redis round-trips each renewal window. With one timer per worker batching through extendLocks-1.lua, the same N jobs cost one timer and one round-trip. Move the slider to read the difference at a given concurrency.

Per renewal window · set the number of held jobs

timer-per-job (naive) — LockManager · one timer per worker 1 timer · 1 call

The renewal cost of the LockManager is flat: one timer and one Redis round-trip no matter how many jobs a worker holds. That is the whole reason it is one timer, not N.

The dives

Three dives build the module — the renewal first, then the recovery of jobs the renewal could not save, then the scheduler and the event stream around the lifecycle.

E2.03.1

One timer, not N

EchoMQ.LockManager's single Process.send_after timer per worker — the ~7500ms tick, the half-window threshold scan, and the batched extend_locks/5 over extendLocks-1.lua.
E2.03.2

Stalled recovery

EchoMQ.StalledChecker — LRANGE active, a pipelined EXISTS-lock probe, the two-phase mark-then-recover sweep, moveStalledJobsToWait-8.lua, and maxStalledCount default 1.
E2.03.3

Schedulers & events

EchoMQ.JobScheduler's every-vs-cron next-run computation, and EchoMQ.QueueEvents consuming the emq:{queue}:events Redis Stream over XREAD as {:echomq_event, type, data}.

How it fits

Every process in this module stands on the two immutable layers E1 froze. The lock key emq:{queue}:{jobId}:lock and the field names are L1; the renew script extendLocks-1.lua, the recovery script moveStalledJobsToWait-8.lua, the scheduler script family, and the emq:{queue}:events stream that every transition script XADDs onto are L2. The four library processes that drive them — LockManager, StalledChecker, JobScheduler, QueueEvents — are L3/L4: each is an OTP GenServer choice, and Go and Node.js make their own. Because EchoMQ-Elixir ships no supervision tree of its own, the host application supervises these four as children. L1 and L2 are byte-identical across runtimes; only L3 and L4 differ — the Golden Rule.

the protocol (immutable L1/L2)

The lock key emq:{queue}:{jobId}:lock, the scripts extendLocks-1.lua and moveStalledJobsToWait-8.lua, and the emq:{queue}:events stream every transition script writes — frozen, the same in every runtime.

its three runtimes (variable L3/L4)

Elixir drives them with four supervised GenServers — one LockManager timer per worker, a StalledChecker sweep, a JobScheduler, an XREAD-consuming QueueEvents; Go and Node.js drive the same scripts and stream their own way. The wire underneath does not move.

The Golden Rule: L1 and L2 are immutable and shared; L3 and L4 vary. The lock key, the renew and recovery scripts, and the event stream are the protocol — the same in every runtime; the processes above them are each runtime's own.

Start with one timer, not N, then read the stalled recovery, then the schedulers and the event stream. This module follows E2.01.3 · the lock protocol and E2.02 · jobs, queues & workers — reachable from E2 · the core.

References

Sources

  • BullMQ — Documentation — the lock-renewal protocol, the stalled-job recovery sweep, repeatable jobs, and the events stream EchoMQ implements.
  • Redis — EXISTS — the lock-presence probe the stalled checker pipelines over the active list.
  • Redis — XADD — the append each transition script makes onto the queue's events stream.
  • Redis — SET — the SET key token PX ms that re-sets a lock's TTL on renewal.

Related in this course