E2.02 · Movement I · The Core
Jobs, queues &
the worker
Three thin components move a job along the lifecycle — and Redis, not the component, holds the state.
The lifecycle (E2.01) is the shape of a job's life; this module is the three
components that move a job through it. A Job (EchoMQ.Job) is a thin
struct over a Redis hash. A Queue (EchoMQ.Queue.add/4) is a string name plus
a connection, with no in-memory store. A Worker (EchoMQ.Worker) is a
GenServer fetch loop bounded by concurrency, running one BEAM process per job. Pull any
component out and Redis still holds every queue's state — that is the module's thesis. The core is shipped, so
every surface below is read from the real source. Three dives follow.
Movement I · Core (as-built) ← redis-patterns R3
Pull a component out
Each component looks like it holds the queue, and none of them does. A Job's fields, a Queue's job list, and a
Worker's in-flight set all live in Redis — the job hash, the emq:{q}:wait LIST, the
emq:{q}:active LIST and lock. Pick a component, then read what survives if its process is
restarted: the durable state is Redis, not the caller.
The three components · pick one, read what Redis owns
A Job, a Queue, and a Worker are all thin views over Redis. Restart any of them and the queue is intact, because Redis — not the component — holds the state.
The dives
Three dives build the module — the Job model first, then the stateless queue that admits it, then the worker fetch loop that runs it.
The job model
EchoMQ.Job.new/4 — a thin struct over a Redis hash; the opts that route a job (delay, priority, parent); the pure state predicates; from_redis/4.
The stateless queue
EchoMQ.Queue.add/4 — two clauses, one stateless path: a string name + a connection, no in-memory store; the add* script per target; what survives a restart.
The worker fetch loop
EchoMQ.Worker — available_slots = concurrency − in-flight; the atomic moveToActive-11 pickup; a BEAM process per job via Task.async; refill on completion.
How it fits
The three components stand on the two immutable layers E1 froze. L1 is the keys and the hash field
names; L2 is the Lua scripts (addStandardJob-9,
moveToActive-11, …). Those do not move across runtimes. L3 is the executor
and L4 the API: EchoMQ.Job, EchoMQ.Queue.add/4, and
EchoMQ.Worker are Elixir's; Go and Node.js drive the same hashes, keys, and scripts their
own way. EchoMQ-Elixir is library-only — it ships no Application or
supervision tree of its own; the host app supervises EchoMQ.Worker and
EchoMQ.RedisConnection. The default key prefix is bull, so a
queue's keys read emq:{queue}:wait, emq:{queue}:active, and so on.
The keys and the hash field names (L1) and the Lua scripts (L2) —
addStandardJob-9, moveToActive-11 — frozen, the same in
every runtime.
Elixir materializes them as EchoMQ.Job, EchoMQ.Queue.add/4, and
the EchoMQ.Worker GenServer; Go and Node.js build their own Job, Queue, and Worker over
the same wire. The components vary; the keys and scripts do not.
The Golden Rule: L1 and L2 are immutable and shared; L3 and L4 vary. The Job, the Queue, and the Worker are each runtime's own materialization of the same protocol.
Start with the job model, then the stateless queue, then the worker fetch loop. The lifecycle these components move a job through is E2.01; the chapter is E2 · the core.
References
Sources
- BullMQ — Documentation — the Job, Queue, and Worker model EchoMQ rides, and the add scripts behind each.
- BullMQ — Home — the project the wire protocol comes from.
- Redis — Lists — the LIST behind the wait and active queues a worker fetches from.
- Redis — RPOPLPUSH — the reliable wait→active move at the heart of the worker pickup.
Related in this course
- E2 · The core — the chapter landing.
- E2.01 · The lifecycle & state machine — the shape this module fills.
- E1 · The protocol — the immutable wire these components ride.
- redis-patterns R3 · Reliable queues — the pattern this module deepens.