jonnifyechomq
EchoMQ / The Protocol / Workshop

The Protocol · workshop · hands-on

Decode a key, trace a script

The protocol is two things you can read by hand: a key, and a script.

This page closes the chapter by doing both on real artifacts — no new surface, only the ones already taught, exercised. Decode a key: given emq:{quotes}:job:JOB0KHTOWnGLuC, read off the queue, compute the cluster slot, and read the kind from the branded id. Trace a script: take the EchoMQ.Jobs @enqueue handle, read its decoded Lua, and point at the keys it declares in KEYS[]. Start with the decoder on the right; type any key.

Decode a key Trace a script Pattern → implementation References

The key decoder · type a key

queue

—

slot (of 16384)

—

kind

—

Decode a key — emq:{quotes}:job:JOB0KHTOWnGLuC

The grammar is fixed by EchoMQ.Keyspace. A per-queue key is the literal emq:, the queue name in a single brace pair, then the type suffix — emq:{q}:<type>. A job row's type is job:<id>, so job_key/2 composes emq:{q}:job:<branded-id>. The id is gated before it is used: an ill-formed id raises and never reaches a key.

# echo_mq — EchoMQ.Keyspace
# A per-queue key is emq:{q}:<type>. A job row's type is "job:" then the
# branded id, so job_key/2 is queue_key(queue, "job:") with the id appended.
# The id is gated first — an ill-formed id raises, never reaches a key.
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

def job_key(queue, branded) when is_binary(branded) do
  if EchoData.BrandedId.valid?(branded) do
    queue_key(queue, "job:") <> branded
  else
    raise ArgumentError, "job_key requires a valid branded id"
  end
end

Read the key in three fields. The queue is the substring inside the first {...} — quotes. The slot is CRC16-XMODEM(hashtag) mod 16384, computed client-side over the hashtag so the connector routes without a server round trip. For quotes the slot is 3378. Because the slot is taken over the hashtag, every key of one queue lands on the same slot — the pending set, the active set, and this job row are all on 3378. The canon vector confirms the algorithm: slot("123456789") == 12739.

# echo_mq — EchoMQ.Keyspace.slot/1 — CRC16-XMODEM over the hashtag, % 16384
# Client-side cluster routing: the connector knows the slot without asking the
# server. Known vector: slot("123456789") == 12739.
def slot(key) when is_binary(key), do: rem(crc16(hashtag(key), 0), 16384)

def hashtag(key) do
  with [_, rest] <- :binary.split(key, "{"),
       [tag, _] when tag != "" <- :binary.split(rest, "}") do
    tag           # the substring inside the first {...}
  else
    _ -> key      # no brace pair → the whole key is its own hashtag
  end
end

The kind is the first three bytes of the branded id. JOB0KHTOWnGLuC is a 14-byte branded id — a three-letter namespace plus an 11-character Base62 payload. EchoData.BrandedId.parse/1 splits it, and namespace/1 reads the first three bytes: the kind. Here it is JOB — the exact value the @enqueue script gates at the wire.

# echo_data — EchoData.BrandedId: a 14-byte id = 3 [A-Z] namespace ++ base62(snowflake) padded to 11.
# parse/1 splits namespace from snowflake; namespace/1 reads the first 3 bytes — the "kind".
iex> EchoData.BrandedId.parse("JOB0KHTOWnGLuC")
{:ok, "JOB", 274557032793636864}
iex> EchoData.BrandedId.namespace("JOB0KHTOWnGLuC")
"JOB"

So the key decodes to queue quotes, slot 3378, kind JOB — a job row on the queue's slot. The decoder above runs these exact steps: pure functions mirror hashtag/1, slot/1, and BrandedId.parse/1 over whatever you type, and a key without the emq:{q}: grammar is named malformed rather than guessed.

A key is legible by hand: the braces give you the queue, a CRC16 over them gives you the slot, and the first three bytes of the id give you the kind.

Trace a script — @enqueue to its declared keys

A verb is a named handle over a Lua script, and the script is the protocol. Read the enqueue transition in two beats. Beat one — the named handle. EchoMQ.Jobs.enqueue/4 builds the two keys the script will touch — the job row and the pending set — and runs the @enqueue handle through the connector. The keys are built host-side, by the keyspace, and handed in declared; the values ride ARGV.

# echo_mq — EchoMQ.Jobs
# Beat one: the named handle. The host builds the two keys the script touches —
# the job row (KEYS[1]) and the pending set (KEYS[2]) — and hands them in
# declared. The connector runs @enqueue EVALSHA-first. The values ride ARGV.
def enqueue(conn, queue, job_id, payload) when is_binary(job_id) and is_binary(payload) do
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case Connector.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}
    {:ok, 0} -> {:ok, :duplicate}
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}
    other -> other
  end
end

The handle resolves to a script. EVALSHA-first means the connector runs the script by its precomputed SHA1, reloading the source once on a NOSCRIPT miss: EchoMQ.Script.new/2 precomputes the SHA, and EchoMQ.Connector.eval/5 dispatches. Beat two — the script body. @enqueue admits by kind, refuses a duplicate, writes the row, and inserts the pending entry — one atomic step, every key declared.

-- the @enqueue script (EchoMQ.Jobs) — one atomic transition, every key declared
-- KEYS[1] = the job row    KEYS[2] = the pending set
-- ARGV[1] = the branded id  ARGV[2] = the payload
-- The branded-id gate: the id must be JOB-namespaced — refused at the wire.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
-- Idempotent: a row that already exists is a no-op (a duplicate enqueue).
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
-- The three-field row: state, attempts, payload — the shape every speaker reads.
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- The pending set: the id is the member at score 0, so byte order is mint order.
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

The script declares two keys — KEYS[1] the job row, KEYS[2] the pending set — and constructs none in-script. That is the law: every key a script touches is passed in KEYS[]; ARGV carries values only. The decoded key from the first move is exactly the KEYS[1] this script writes: emq:{quotes}:job:JOB0KHTOWnGLuC is the job row, on slot 3378, with the kind JOB the gate admits.

The script tracer · pick a verb

enqueue handle @enqueue KEYS[] KEYS[1] KEYS[2]

The tracer reads a different thing from the decoder: the decoder reads one key, the tracer reads the whole set of keys a transition touches. Both come straight off real artifacts — the handle's declared keys are the contract the script runs under.

A handle resolves to a script; a script names its keys. Trace any verb and you have the keys it touches and the order it touches them in.

Pattern → implementation

Decoding a key and tracing a script is reading the protocol with no runtime in the way — the move the Redis Patterns Applied course frames, applied here in depth.

the pattern (Redis Patterns Applied)

A Valkey convention becomes a protocol when the agreement is pushed below the line — the keys plus the atomic scripts — and read directly off the wire. Patterns become protocol teaches that move.

the implementation (echo_mq)

EchoMQ.Keyspace.queue_key/2 fixes emq:{q}:<type> and slot/1 reads the routing off the key; the EchoMQ.Jobs @enqueue script fixes the transition with every key declared in KEYS[]. The wire, read by hand.

The protocol is legible: a key gives you the queue, the slot, and the kind; a handle gives you the declared keys and the atomic order. Read one of each and you have read the wire.

Recap — the chapter, exercised

The chapter taught the owned keyspace, the record hash, the Lua layer, and immutability with branded ids. This close exercised them on real artifacts: emq:{quotes}:job:JOB0KHTOWnGLuC decodes to queue quotes, slot 3378, kind JOB; the EchoMQ.Jobs @enqueue handle resolves to a script declaring KEYS[1] the row and KEYS[2] the pending set. The next step is the chapter map — the three pillars that all stand on this substrate.

Back to The Protocol — the owned keyspace, the record hash, the Lua layer, and immutability with branded ids, and the door onward to the three pillars.

References

Where the keyspace, the slot algorithm, and the scripting dispatch are grounded.

Sources

  • Valkey — Documentation — the BSD-licensed, foundation-governed store EchoMQ is backed by; the substrate of record.
  • Valkey — EVALSHA — the load-once, run-by-SHA dispatch the connector uses to run the @enqueue script.
  • Redis — Keyspace & hash tags — cluster key routing by the hashtag inside {...}, the mechanism the {q} brace and slot/1 use.
  • Redis — ZADD — the same-score sorted-set insert @enqueue uses for the pending set, so byte order is mint order.

Related in this course