ZRANGEBYSCORE promotion

The read is a range query bounded above by the current clock. Every member from the start of the set up to now is due.

ZRANGEBYSCORE delayed_queue -inf <now> LIMIT 0 10   # the due head, earliest first

-inf is "no floor" — start from the lowest score. The upper bound is the current time. LIMIT 0 10 caps the batch to ten so one sweep claims a bounded slice and a backlog drains over several sweeps rather than one giant move. The result is the due head: the members whose fire-time has passed, earliest first. The range is a read only — it names the due jobs but does not remove them. Between the read and the removal another worker can read the same members, which is why the claim must be atomic and separate.

Each due member is claimed with a ZREM. The removal is the claim.

ZREM delayed_queue "task:abc123"   # 1 if this caller promoted it, 0 if another already did

ZREM returns 1 when this call removed the member — the caller owns the task — or 0 when another worker already removed it. A worker proceeds with a task only when its ZREM returns 1. With many workers sweeping the same set, the range may hand the same member to several of them, but only one ZREM returns 1; the rest get 0 and drop it. The removal establishes the owner, atomically, with no extra lock.

The range and the ZREM are two commands, and between them a second worker can interleave. To make poll-and-claim a single atomic step, run both inside a Lua script.

local tasks = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, ARGV[2])
for i, task in ipairs(tasks) do
    redis.call('ZREM', KEYS[1], task)
end
return tasks

The script reads the due head and removes every member in one round trip; no other worker can claim a job between the range and the ZREM. This is the shape EchoMQ's sweep takes — a range, a ZREM, and a move to the target list, all inside one script, on the worker path.

In EchoMQ the sweep is promoteDelayedJobs, an included function — an identical copy compiled into moveToActive-11.lua, retryJob-11.lua, and moveToFinished-14.lua. It is not a standalone script and is never called as one; it runs as part of the scripts the workers already execute, so a due delayed job is promoted on the path of normal work, not by a separate poller. Its body is the range-then-claim, with the shifted upper bound the fire-time score requires.

-- promoteDelayedJobs (included in moveToActive-11 / retryJob-11 / moveToFinished-14) — the due-sweep (real)
local jobs = rcall("ZRANGEBYSCORE", delayedKey, 0, (timestamp + 1) * 0x1000 - 1, "LIMIT", 0, 1000)
if #jobs > 0 then
  rcall("ZREM", delayedKey, unpack(jobs))            -- claim the whole due head atomically
  for _, jobId in ipairs(jobs) do
    -- priority 0 → LPUSH onto the wait list; priority > 0 → ZADD onto the prioritized set
  end
end

The upper bound is (timestamp + 1) × 0x1000 - 1, not timestamp, because the score is the fire-time shifted twelve bits: that bound is the top of the current millisecond's band, so the range catches every job due in this millisecond including its within-millisecond tie-breaks. The LIMIT 0 1000 caps each sweep at a thousand jobs. Each promoted job goes to the wait list when its priority is 0, or onto the prioritized set when it carries a priority — the priority reading a later module takes up. A distinct script promotes a single named job on demand: EchoMQ.Scripts.promote/3 runs promote-9.lua, which ZREMs one job from the delayed set (KEYS[1]) and moves it to the wait list (KEYS[2]). The bulk sweep moves the whole due head by score; promote-9.lua moves one job by name, now.

The range query that reads the due head, the ZREM that claims each member, the sorted set behind the sweep, and the queue protocol EchoMQ ports.