jonnifyechomq

The lifecycle · dive 01

The four sets

A job's whole position in its life is which sorted set its id is a member of.

There are four — pending, active, schedule, dead — and the member is the branded id itself, not a pointer to it. Because the id is a branded Snowflake (a JOB namespace over a time-ordered 11-char Base62 part, 14 bytes), byte order is mint order — so a set ordered lexically by member is a queue ordered oldest-first by construction, with no second index to maintain. Pick a set on the right, then read how browse/3 and pending_size/2 read it.

The keys Browse needs no index Inspect the sets Pattern & implementation References

The four sets · pick one

pending emq:{q}:pending score 0 · mint order active emq:{q}:active score = lease deadline schedule emq:{q}:schedule score = run-at instant dead emq:{q}:dead score 0 · the morgue

The keys are built by the keyspace, not the data

Every set key comes from EchoMQ.Keyspace.queue_key/2, which composes emq:{ + the queue + }: + the type. The per-queue {q} is a cluster hashtag, so all four of one queue's sets land on one slot — which is what lets one Lua script touch them in a single atomic transition. The row key is the same builder, and it gates the id before composing: a malformed id raises before it ever reaches the wire.

# echo_mq — EchoMQ.Keyspace
# queue_key builds emq:{q}:<type>. The braces are a cluster hashtag: every key
# of one queue hashes on the {q} substring, so pending / active / schedule / dead
# all land on ONE slot — which is what lets one Lua script touch them atomically.
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

# job_key gates the id BEFORE it ever reaches a key. A malformed id raises here,
# never reaching the wire — the row key cannot be forged from bad data.
def job_key(queue, branded) when is_binary(branded) do
  if EchoData.BrandedId.valid?(branded) do
    queue_key(queue, "job:") <> branded   # emq:{q}:job:<id>
  else
    raise ArgumentError, "job_key requires a valid branded id"
  end
end

The grammar is emq:{q}:job:<id> for the row and emq:{q}:<type> for the four sets. The id is the member of a set; the same gated id composes the row key. One queue, one slot, one grammar.

A job's state is not a column to read — it is a set to be a member of. The keyspace builds the four addresses; the id is what lives at them.

Mint order is byte order, so browse needs no index

Because the member is the id, and the id sorts by its own bytes (mint order), the newest-first browse is a plain lexical range over the set — reverse, by member, no score sort and no auxiliary structure. The depth is one count. The set already holds the answer; the read only asks it.

# echo_mq — EchoMQ.Jobs
# browse the pending set newest-first. The members ARE the ids, ordered by their
# own bytes (mint order), so ZRANGE ... REV BYLEX walks newest-to-oldest with no
# second index. + and - are the lexical max / min bounds.
def browse(conn, queue, n) when is_integer(n) and n > 0 do
  key = Keyspace.queue_key(queue, "pending")
  Connector.command(conn, ["ZRANGE", key, "+", "-", "BYLEX", "REV", "LIMIT", "0", Integer.to_string(n)])
end

# depth is one count. ZCARD over the pending set — no scan, no index.
def pending_size(conn, queue) do
  Connector.command(conn, ["ZCARD", Keyspace.queue_key(queue, "pending")])
end

browse/3 returns the n newest pending ids; pending_size/2 returns the pending depth. Neither maintains anything — there is no "newest" index to keep in step with the set, because the set's own member order is the order.

A second index is a thing that can drift. By making the id the member and the bytes the order, the queue has nothing to keep in sync — the one structure is both the membership and the order.

Inspect the sets

Here is a fixed pending set of five jobs, in mint order. Run browse/3 for a given n and read the actual ZRANGE … REV BYLEX slice it returns (newest first), and the ZCARD the depth reads. The model is the real read logic over this fixed set — pure, no network.

browse/3 over a fixed pending set · choose n

3
emq:{q}:pending (ZSET, score 0) — mint order, oldest left ·

The browse takes the newest n from the high end of the lexical order (REV from +). Nothing was sorted at read time — the set was already in mint order because the members are the ids.

Pattern & implementation

the pattern (Redis Patterns Applied)

A job's state is encoded by which collection it lives in, not a status column — so a move between states is a move between sets. States as locations teaches it; Reliable Queues is the chapter.

the implementation (echo_mq)

Four sorted sets keyed by queue_key/2, members the branded ids, so byte order is mint order and browse/3 + pending_size/2 read the set with no second index.

The pattern says: let the location be the state. The implementation makes the member the id too — so the one structure carries the membership and the order.

Recap — the set is the state, the id is the member

A job lives in one of four sorted sets, and its id is the member. The branded id's bytes are mint order, so the set is oldest-first by construction; browse/3 reads it with ZRANGE … REV BYLEX and pending_size/2 with ZCARD, neither needing a second index. The next dive reads how a job leaves the pending set: the claim, the lease, and the fencing token.

Next in the lifecycle: Claim & the lease — the @claim transition, the active-set score that IS the lease deadline, and attempts as the fencing token.

References

Where the set-as-state and the index-free browse are grounded, and the commands the reads use.

Sources

Related in this course