The Queue · pillar I
Distribute work
over the wire
Enqueue, claim, complete — one job, one worker, a state machine over four sorted sets.
The Queue is the first pillar: a producer enqueues a job, one worker
claims it, runs it, and completes it — and a failure retries with backoff or
dies into a morgue. It is all real code. The machine is four sorted sets
(pending, active, schedule,
dead) where the active-set score is the lease and the row's
attempts is the fencing token — no separate lock. Above it sit fair
lanes over a rotating ring, a consumer loop that parks instead of polling, bulk
batches, the lifecycle controls, and parent/child flows.
The life of a job
A job is one row and a place in one of four sorted sets. Every verb of the Queue moves it between those places. Pick a state to read what it means, the set it lives in, and where it goes next — then take the module that owns those transitions.
A job's state machine · pick a state
One row, four sets: claim leases it, complete retires it, retry reschedules or kills it, and reap recovers the lease a dead worker never returned.
The modules
Five modules, each a hub and three dives, then a workshop. Each grounds in the real
echo/apps/echo_mq code — the state machine, the lanes, the consumer loop, the batch
path, the control plane, and the flow orchestration.
The lifecycle
The state machine: four sorted sets, the lease that is the active score, attempts as the fencing token.
Jobs, lanes & the consumer
The idempotent producer, fair lanes over a rotating ring, and the loop that parks instead of polling.
Batches
Bulk enqueue, bulk flows, and batch lease extension — many at once in one wire flush.
Lifecycle controls
Schedule, backoff, repeat; cancel and checkpoint; the stalled sweep; the operator's queue-scope plane.
Flows
Parent/child orchestration: the atomic same-queue fan-in, cross-queue delivery, and the failure policy.
Workshop
Trace one job across the four sets, then compose a flow and watch the fan-in release the parent.
Redis Patterns Applied
This is the depth behind the Redis Patterns Applied chapters that door here: R3 · Queues (the reliable-queue, ack, and visibility patterns this state machine makes concrete) and R4 · Time, delay & priority (the delayed and scheduled jobs the schedule set carries). The orchestration patterns of R6 · Flow control land here too, in the Flows module. There the pattern is the door; here is the wire.
The rest of the system
The Queue is one reading of the shared substrate. The Bus broadcasts signals across the same wire; the Cache serves reads in front of it; the Proof chapter shows the whole system holds. The Protocol below them all is ready to read now.
The Protocol
The substrate the Queue runs on: the keyspace, the record, the Lua layer, immutability.
The Bus
Broadcast signals: events, the event log, consumer groups, time-travel.
The Cache
Serve reads: cache-aside, single-flight, coherence on the Bus.
The Proof
The system holds: the conformance suite, telemetry, the benchmark gate.
The Queue and the Protocol are ready to read; the Bus, the Cache, and the Proof are on the build front, each added as its pages land.
References
Sources
- Valkey — Documentation — the store the sets, the rows, and the scripts run inside.
- Valkey — ZPOPMIN — the oldest-pending pop the claim is built on.
- Valkey — LMOVE — the rotate-one-step the fair-lane ring uses.
- Valkey — BLPOP — the park the consumer loop blocks on instead of polling.
- Redis — EVALSHA — the load-once, run-by-SHA dispatch every transition uses.
Related in this course
- The Protocol — the substrate the Queue runs on.
- Overview — the chapter that frames the three pillars.
- redis-patterns R3 · Queues — the reliable-queue patterns, the near side of the door.
- The Branded Component System — the architecture the wire belongs to.