R4.01 · Time, Delay & Priority
The delayed queue
Schedule tasks for future execution with a sorted set whose score is the time the task should run. Lists deliver FIFO but cannot defer; a sorted set gives time-ordering and a range query for the due head.
A delayed queue answers one need: run this task later, not now. A Redis list cannot express it
— a list delivers in the order it was filled, head to tail, with no notion of when a member becomes eligible.
A sorted set can. Score each task by the time it should run, and the set orders its members by fire-time for free;
a range query bounded by the current clock returns exactly the tasks whose time has come. This module takes the
textbook pattern apart and then shows the one move EchoMQ sharpens: scoring by timestamp ×
0x1000 so the low twelve bits discriminate jobs due in the same millisecond.
Data Model
The structure is a single sorted set. Each member is a task identifier; each score is the time the task should run.
- Member — the task identifier or payload reference.
- Score — the timestamp at which the task should execute.
Tasks sort themselves by execution time, earliest first, with no extra index. The score is the whole ordering: read the set low-to-high and the earliest-due task is at the head. The interactive below loads a fixed set of six jobs scored by fire-time, sweeps the clock forward, and reports which jobs fall inside the due range and which job is the next-wake head.
The delayed set · six jobs scored by fire-time, swept by the clock
A delayed set orders its members by fire-time for free; the score is the whole index, and reading the head gives the earliest-due job.
Scheduling a Task
To schedule a task to run at a future time, add it to the set with that time as the score. The score is the moment the task should run.
ZADD delayed_queue 1706649000 "task:abc123" # score = the fire-time
The payload can live separately, with the set holding only the reference — the set carries the schedule, a hash carries the data.
HSET task:abc123 type "send_email" recipient "user@example.com" ZADD delayed_queue 1706649000 "task:abc123"
One ZADD schedules a task: the member is the task, the score is when it
runs; the payload lives in a separate hash.
Polling for Ready Tasks
A worker reads the tasks whose time has passed with a range query bounded above by the current clock. Every member from the start of the set up to now is due.
ZRANGEBYSCORE delayed_queue -inf 1706648500 LIMIT 0 10 # the due head, up to ten
This returns up to ten tasks with scores at or below the current time. The -inf lower
bound means "no floor" — from the start of the set up to the clock. The LIMIT 0 10
caps the batch so one poll claims a bounded slice. The first dive scores the jobs this poll reads; the second
dive runs the sweep that claims them.
The poll is a range query bounded by the clock: ZRANGEBYSCORE -inf now
returns the due head, capped by LIMIT.
Claiming Tasks Atomically
Many workers may poll the same set at once, so a worker must claim each task atomically before processing it,
or two workers process the same task. The claim is a ZREM.
ZREM delayed_queue "task:abc123" # returns 1 if this caller won the claim, 0 if already gone
ZREM returns 1 when this call removed the member — the
caller won the claim — or 0 when another worker already removed it. A worker
processes a task only when its ZREM returns 1. The removal is
the claim; nothing else marks ownership.
The ZREM is the claim: a 1 means this worker owns
the task, a 0 means another already took it.
The Complete Flow
The four steps are poll, claim, process, clean up — a range query to read the due head, an atomic
ZREM per task, the work, and a delete of the payload.
| Step | Command |
|---|---|
| Poll | ZRANGEBYSCORE delayed_queue -inf <now> LIMIT 0 10 |
| Claim | ZREM delayed_queue "task:abc123" per task |
| Process | run the task only if its ZREM returned 1 |
| Clean up | DEL task:abc123 |
Poll, claim, process, clean up — the range reads the head, the ZREM
claims each, and only a winning claim runs the task.
Avoiding Busy Waiting
Polling an empty set on a fixed tick wastes work. The avoidance is to read the head and sleep until it is due.
ZRANGE delayed_queue 0 0 WITHSCORES # the earliest task and its scheduled time
This returns the earliest task and its scheduled time. When nothing is due yet, a worker sleeps until that time, or a bounded maximum interval, which cuts polling overhead sharply when the set is empty or the next task is far out. The next-wake time is the head's score. The third dive reads that head and carries it on a marker so a blocked worker wakes exactly when due.
The head's score is the next wake; sleeping to it instead of polling on a tick makes an idle set nearly free.
Combining with Lua
The poll and the claim are two commands, and between them another worker can claim the same task. To make
poll-and-claim one atomic step, run both inside a Lua script — a range query, then a
ZREM per member, in one round trip.
local tasks = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, ARGV[2]) for i, task in ipairs(tasks) do redis.call('ZREM', KEYS[1], task) end return tasks
The script finds and claims the ready tasks in a single round trip; no second worker can interleave between the
range and the removal. This is the shape EchoMQ's sweep takes — a range followed by a
ZREM, inside one script, on the worker path. The second dive walks the real
promoteDelayedJobs function it becomes.
A Lua script folds the range and the ZREM into one atomic poll-and-claim, so
no worker interleaves between reading the due head and removing it.
When to Use / When to Avoid
A failed task reschedules itself by re-adding with a future score — a retry is a delayed task like any
other (ZADD delayed_queue <now + 60> "task:abc123"). The delay can grow with the
attempt count, but that math is a module of its own: R4.04 · Backoff and retry owns the
formula (EchoMQ.Backoff.calculate/4, base × 2^(n-1)).
This module schedules the retry; the formula that chooses the delay is the next reading of the chapter.
The sorted-set delayed queue fits scheduled work where the ordering axis is time and the read is a due-range: scheduled and deferred work (send at a time, notify after a delay, handle off-peak), retry with backoff, and rate spacing. It fits less well where the queue needs consumer groups, message acknowledgement, or replay — those are Redis Streams territory. A sorted set delivers time-ordering and an efficient due-range query, not an acknowledgement protocol.
| Use it for | Reach elsewhere when |
|---|---|
| Scheduled / deferred work: send at a time, notify after a delay, off-peak handling | The queue needs consumer groups or message acknowledgement — use Redis Streams |
| Retry with backoff: reschedule a failed task with a growing delay | The work needs replay of an acknowledged history — a stream's log |
| Rate spacing: space calls over time by their fire-time | The ordering axis is priority, not time — the prioritized set, a later module |
Reach for the delayed queue when the ordering axis is time and the read is a due-range; reach for a stream when the queue needs groups, acknowledgement, or replay.
The three dives
Each dive takes one layer of the delayed queue, building on the source spine and adding the EchoMQ refinement on top: the score that orders the set, the sweep that promotes the due head, and the wake that avoids polling an empty set. Read them in order — the score first, the sweep second, the wake third.
The score is the fire-time
The textbook scores by a raw Unix timestamp; EchoMQ scores
delayedTimestamp × 0x1000, so the low twelve bits discriminate jobs due in the
same millisecond.
ZRANGEBYSCORE promotion
The due-sweep: range from 0 to now, ZREM the head,
move it to the wait list — EchoMQ's included promoteDelayedJobs function.
The next wake
Not busy-polling an empty set: read the head, compute the next due time, and let the delay marker
(ZADD markerKey nextTimestamp "1") wake a blocked worker exactly when due.
A delayed queue scores each task by the timestamp at which it should run, and reads the due head with a range query bounded by the current clock.
EchoMQ.Keys.delayed/1 → emq:{queue}:delayed scores by
(timestamp + delay) × 0x1000 in addDelayedJob-6.lua's
getDelayedScore, so the low twelve bits discriminate jobs due in the same
millisecond; a promoteDelayedJobs sweep moves the due head to the wait list.
A delayed queue is a sorted set scored by fire-time; EchoMQ scores ts ×
0x1000 so a millisecond can hold more than one ordered job, and recovers the real time by dividing the
shift back out.
A door, not a depth
This module cites one excerpt of EchoMQ's protocol — the getDelayedScore and
promoteDelayedJobs functions in addDelayedJob-6.lua and the
worker scripts — as proof the delayed score ships. The full scheduler subsystem, the delay /
promote / retry coordination across a worker pool, and the polyglot concurrency models are the subject of the
dedicated EchoMQ course, built next with this toolkit. From here, open onto
E6 · Lifecycle controls. This module teaches the delayed queue; that
course teaches the scheduler that runs it.
After R4.01, the Time, Delay & Priority chapter continues with the scheduler registry, the composite priority score, backoff-and-retry, and leaderboards, closing with the EchoMQ scheduler workshop. Return to the Time, Delay & Priority chapter to continue.
References
The Redis commands behind the delayed queue — the sorted set as a timer wheel, the schedule, the due-range read, and the atomic claim — and where the scheduler that runs them is taught.
Sources
- Redis — Sorted sets — the sorted set as a timer wheel: members ordered by a numeric score, read by score range.
- Redis — ZADD — score a member by its fire-time; the single write that schedules a delayed task.
- Redis — ZRANGEBYSCORE — the range query that reads the due head, the read inside the sweep.
- Redis — ZREM — the atomic claim:
1means this caller won the task,0means another already did. - BullMQ — the queue protocol — the delayed set and marker protocol EchoMQ ports, where the Lua scripts are the protocol.
Related in this course
- R4.01.1 · The score is the fire-time — the fire-time score, shifted twelve bits.
- R4.01.2 · ZRANGEBYSCORE promotion — the due-sweep to the wait list.
- R4.01.3 · The next wake — read the head, wake on the marker.
- R4 · Time, Delay & Priority — the chapter.
- R4 · The sorted set as a clock — the orientation dive: one set, two readings.
- R3 · Reliable Queues — the chapter this one feeds.
- E6 · Lifecycle controls — the dedicated EchoMQ course: the delay / promote / retry scheduler in depth.