Packing two keys in one score

A sorted set offers one ordering axis — the score. A priority queue needs two. Jobs in a higher-priority tier must come out before jobs in a lower tier, and within one tier the earliest-arriving job must come out first. A single number has to express both.

The move is positional. Reserve the high bits of the score for the tier and the low bits for the arrival order. Then the score compares tier-first automatically: any difference in the high bits dominates any difference in the low bits, because the high bits are worth more. Comparing two composite scores compares their tiers first and, only when the tiers are equal, their arrival counters.

EchoMQ computes the score with getPriorityScore, an included Lua function whose canonical home is addPrioritizedJob-9.lua.

-- getPriorityScore.lua (included by addPrioritizedJob-9) — real
local function getPriorityScore(priority, priorityCounterKey)
  local prioCounter = rcall("INCR", priorityCounterKey)
  return priority * 0x100000000 + prioCounter % 0x100000000
end

0x100000000 is 232 — 4294967296. Multiplying the priority by it shifts the tier left thirty-two binary places, leaving the low thirty-two bits at zero. The arrival counter, taken % 232, then fills those low bits. The result is one number whose top half is the tier and whose bottom half is the position within the tier.

One IEEE double carries the whole thing exactly. A double holds fifty-three bits of integer precision; a small priority shifted into the high thirty-two bits plus a counter in the low thirty-two stays well under that ceiling, so the score is exact and the ordering never rounds.

Because the tier sits in the high bits, any change in the tier outweighs every value the counter can take. Two jobs in tier 1, however far apart their counters, both score below any job in tier 2 — the smallest tier-2 score is 2 × 232, larger than the largest tier-1 score, which is 1 × 232 + (232 − 1). The high bits set the band; the low bits position within it.

The hero figure above shows it directly: raise the priority slider by one and the score jumps by a full 0x100000000, dwarfing any counter move. The interactive below makes the comparison explicit, pitting two jobs against each other — one a step higher in tier, the other far ahead on the counter — and reporting which composite score is smaller and therefore pops first.

The formula makes a higher-precedence job a smaller number: priority 1 produces a smaller high-bit field than priority 2, so its composite score is smaller. The consume command is ZPOPMIN, which returns the smallest score first. The two compose into the convention a queue user sees: a lower priority number means higher precedence. Priority 1 beats priority 2 because 1 makes the smaller score and ZPOPMIN pops the smallest. The third dive walks that consume in real code.

The whole score is real code in echo/apps/echomq. The prioritized set is EchoMQ.Keys.prioritized/1 → emq:{queue}:prioritized. The arrival-counter key is EchoMQ.Keys.pc/1 → emq:{queue}:pc. The public entry EchoMQ.Scripts.add_prioritized_job/4 runs addPrioritizedJob-9.lua, passing Keys.pc(ctx) as KEYS[9]; the script calls getPriorityScore and then rcall("ZADD", prioritizedKey, score, jobId).

-- addPrioritizedJob-9.lua — score the job, then add it (real)
local score = getPriorityScore(priority, priorityCounterKey)   -- priority × 0x100000000 + INCR pc
rcall("ZADD", prioritizedKey, score, jobId)             -- place the job at its composite score

The same getPriorityScore is reused verbatim across the priority-aware scripts, so every path that scores a prioritized job packs the tier and counter the same way. The number chosen to carry the meaning is the whole pattern: the bare priority says only "this tier"; the composite says "this tier, and this position within it".

The write that scores a prioritized job by its composite number, the sorted set behind it, the atomic counter folded into the low bits, and the queue protocol EchoMQ ports.