R3 · Reliable Queues
Reliable queues
The pivot of the course: a job is never lost — moved in flight under a lock, delivered once that counts, reclaimed if its worker dies.
A reliable queue is not one pattern but a family: an in-flight list under a lock, at-least-once delivery to an idempotent consumer, recovery of a job whose worker crashed, the whole lifecycle moved atomically, and pickup without busy-polling. This is the densest real grounding in the course — drawn straight from EchoMQ's worker fetch loop, where RPOPLPUSH wait→active parks a job under a lock, de:{id} makes a redelivery harmless, and moveToFinished-14 completes the lifecycle in one move. It stands on R2's atomic moves and lock lease.
Overview
A naive queue is a list: push on one end, pop on the other. It loses jobs. The moment a worker pops a job and then crashes — before the work is done, before it is acknowledged — that job is gone from Redis and gone from memory. Nothing reclaims it.
The reliable-queue family is the set of techniques that close every one of those gaps: where the job sits while in flight, how a duplicate delivery is made harmless, how a dead worker's job comes back, how the whole lifecycle moves without tearing, and how a worker waits for work without burning CPU.
EchoMQ implements every one of these in real code, and its worker fetch loop is the worked example for the
whole chapter. moveToActive-11.lua
parks a job in emq:{queue}:active with RPOPLPUSH;
moveToFinished-14.lua
is the whole lifecycle in one EVALSHA; and BZPOPMIN on
emq:{queue}:marker wakes a blocked worker instead of polling.
The reliable lifecycle · wait → active → done, with reclaim
A reliable queue never holds a job in a place where a crash can erase it — every state is a Redis location, and every move between them is recoverable.
Why & when
Use the reliable-queue family whenever losing a job is not acceptable and a worker can crash at any instant — which is every real background-job system. Each failure mode below has one matching technique, and the chapter is the set of those answers.
A worker crashes mid-job
The job must not vanish when a worker dies — move it into an in-flight list under a lock, and reclaim it when the lock expires.
A job runs twice
At-least-once delivery means a job can arrive more than once — make the consumer idempotent and mark completion with de:{id}.
A multi-step transition tears
A lifecycle move touches many keys and must be all-or-nothing — run the whole transition as one Lua script.
The worker busy-polls the queue
A worker that loops on the queue burns CPU and adds latency — block on a marker and let Redis wake it.
Every technique in this chapter answers one of four failures of a naive queue: a lost in-flight job, a double run, a torn transition, and a busy-poll.
The patterns
Three deep dives carry the chapter's orientation — the reliable-queue family in one place, the lifecycle as one atomic move, and where the queue goes next. Each is a full lesson grounded in EchoMQ's worker path. The granular module ladder below takes each pattern in depth, one hub plus three dives per module.
The reliable queue
The family in one place: RPOPLPUSH wait→active under a lock, at-least-once + de:{id} idempotency, and stalled reclaim via moveStalledJobsToWait.
States as locations
The lifecycle as one atomic Lua move: states are Redis locations, moveToFinished-14.lua is one EVALSHA, and :marker + BZPOPMIN replaces busy-polling.
The road ahead
Where the queue goes next: the arc R3→R8, and the door into the living EchoMQ course.
The granular modules — each a hub plus three dives, closing with the workshop capstone:
Processing list
Move a job into an in-flight list so a crash is recoverable: the wait/active LISTs, the atomic RPOPLPUSH/LMOVE transfer, and the in-flight list as the recovery ledger.
At-least-once
Delivery guarantees and why consumers must be idempotent: at-least-once semantics, idempotent consumers, and why exactly-once is a lie — grounded in producer-side de:{id} dedup.
Stalled recovery
Reclaim jobs whose worker died: lock-expiry detection, two-phase mark/recover, and the atomic moveStalledJobsToWait Lua sweep versus the non-atomic Go loop.
Atomic state machine
The whole lifecycle as one atomic Lua move: states are Redis locations, and moveToFinished-14.lua transitions a job — drop the lock, record the result, fetch the next — in one EVALSHA.
Blocking vs polling
Stop busy-polling the queue: the busy-poll cost, the blocking pop, and the marker wake-up — a worker blocks on emq:{queue}:marker with BZPOPMIN and wakes when a job is added.
Workshop
The capstone: a reliable Portal enrollment-job queue assembling R3.01–R3.05 — at-least-once delivery plus an idempotent enroll/2 is exactly-once-in-effect.
How to apply
The hard part is matching the reliable-queue technique to the failure you cannot allow. Name the failure, and the technique — and the real EchoMQ artifact that implements it — follows.
Name the failure · get the reliable-queue technique
There is no single reliable-queue trick — only the technique that closes the failure you cannot accept, each one a real move in EchoMQ's worker loop.
The road ahead
R3 is the spine the rest of the course hangs from. With a reliable queue in hand, each later chapter is a different surface over the same jobs — scheduling, observability, flow control, the data model, and operating the tier. They build on the queue rather than replacing it.
R4 · Time, Delay & Priority
The sorted set as a clock — schedule and prioritize the jobs R3 makes reliable.
R5 · Streams & Events
The durable log — observe the lifecycle R3 defines, replayable after the fact.
R6 · Flow Control
Staying stable under load — rate-limit, batch, and bound concurrency over R3's jobs.
R7 · Data Modeling
How data lives in RAM — the job record and read-models behind the queue.
R8 · Production & Operations
Running the tier at scale — operate everything above in production.
Time, streams, flow control, the data model, and operations are all surfaces over the one reliable queue R3 builds.
Up next
With a reliable queue built, the parallel surfaces over its jobs can follow — time and priority first, then the durable event log and flow control.
The door
This chapter teaches the reliable-queue patterns and proves each with one real EchoMQ excerpt. The engine that runs them — the full worker fetch loop, the heartbeat manager, the stalled-check coordination across a worker pool, and the polyglot concurrency models (a BEAM process pool versus a goroutine semaphore) — is the subject of the dedicated EchoMQ course.
→ EchoMQ. The EchoMQ course is the living companion depth course: it teaches the protocol in depth and tracks the EchoMQ build rung by rung (E1–E8 ⇄ emq.1–emq.6). Per its cross-link map, R3 opens onto E2 · The lifecycle, components & runtimes — the as-built library and the worker fetch loop — E5 · Batches, and E6 · Lifecycle controls.
R3's patterns are transferable Redis techniques; the EchoMQ course owns the implementation depth behind them.
References
Sources
- Redis — RPOPLPUSH — the atomic move that parks a job in the in-flight list.
- Redis — BZPOPMIN — the blocking pop that wakes a worker instead of polling.
- Redis — Scripting with Lua — the atomic lifecycle transition via EVAL / EVALSHA.
- BullMQ — The reliable-queue protocol — the protocol EchoMQ ports across runtimes.
Related in this course
- R2 · Coordination — the atomic moves and lock lease R3 builds on.
- R2.01 · Atomic updates — every state move as one Lua script.
- R0.2 · Redis under Portal — the EchoMQ bus these patterns ground in.
- /elixir · Commands, queries & events — the engine the atomic moves run inside.