The protocol below the line

One line. Below it the keys and the Lua are shared and fixed; above it, the language is yours.

The three pillars interoperate, and any runtime can drive them, for one reason: the protocol lives below the language line. Read EchoMQ as a stack — L0 Valkey, L1 the keys and field names, L2 the atomic Lua, L3 the executor, L4 the API. The shared line falls between L2 and L3. Pick a layer on the right to read its scope and whether it is shared. This dive draws the line on a real key and a real script; the full key taxonomy and the script inventory are The Protocol.

The stack The line, on a real key The line, on a real script Change a field, who breaks? References

The layer stack · pick a layer

The stack, and where the line falls

What makes one wire serve three pillars, in any language? A line. EchoMQ reads as a stack. L0 is Valkey itself — the engine the system runs on, but does not contain. L1 is the data layer: which key holds which structure, and what the field names are. L2 is the set of atomic Lua scripts that move a record between states. Together, L1 and L2 are the protocol — a Valkey data convention plus the scripts that operate on it.

L3 is the script executor: the code that loads each script once and runs it by SHA with EVALSHA, reloading on a NOSCRIPT miss. L4 is the language API a caller invokes — EchoMQ.Jobs.enqueue/4 in the canonical Elixir. L3 and L4 are where a runtime is free to differ. The whole point of the model is that the freedom is confined to those two layers: the shared line falls between L2 and L3.

L4 · API: The language method a caller invokes. Method names, parameter order, and return types vary per runtime.
L3 · executor: The code that loads scripts and runs them by SHA. Connection pooling and the EVALSHA/NOSCRIPT cache vary per runtime.
L2 · Lua: The atomic scripts that change record state. Every key declared in KEYS[]. Shared by everything that speaks the wire.
L1 · keys: The keys (emq:{q}:<type>) and the field names (state, attempts, payload). One agreed convention. Shared.
L0 · Valkey: The engine itself. External — standalone or Cluster.

This dive draws the line on one key and one script. The full key taxonomy and the Lua inventory are the subject of The Protocol — the next chapter, on the course map.

The line, on a real key

The data layer is concrete: a real key the system builds for every queue. EchoMQ.Keyspace.queue_key/2 composes the per-queue key emq:{q}:<type> — the literal emq:, the queue name wrapped in braces, then the type suffix.

# echo_mq — EchoMQ.Keyspace
# Per-queue keys are emq:{q}:<type>. The braces are not decoration —
# they are the cluster hashtag, and they pin every key of one queue to
# one slot.
def queue_key(queue, type) when is_binary(queue) and is_binary(type),
  do: IO.iodata_to_binary(["emq:{", queue, "}:", type])

The {q} is the hashtag. A Valkey cluster routes a key by a CRC16 of the substring inside the first {...}, so wrapping the queue name pins every key of that queue — emq:{orders}:pending, emq:{orders}:active, emq:{orders}:job:<id> — to one slot. One atomic script can touch the pending set, the active set, and the job row in a single round because they are guaranteed co-resident. That key format is L1: the same wherever the protocol runs. Change the literal emq:, or move the braces, and you are speaking a different wire.

The line, on a real script — two beats

The state transitions are L2. They are not application code that happens to touch Valkey — they are server-side Lua that runs inside the engine, atomically. The discipline that makes them portable: every key a script touches is declared in KEYS[]; ARGV carries values only. Read the enqueue transition in two beats.

Beat one — the named handle. EchoMQ.Jobs.enqueue/4 builds the two keys it will touch and runs the @enqueue script through the connector. The keys are built host-side, by the keyspace; the script receives them declared.

# echo_mq — EchoMQ.Jobs
# The host builds the two keys the script touches — the job row and the
# pending set — and hands them in declared. The connector runs @enqueue
# EVALSHA-first (load-on-NOSCRIPT once). KEYS = [row, pending]; the values
# (the id and the payload) ride ARGV.
def enqueue(conn, queue, job_id, payload) when is_binary(job_id) and is_binary(payload) do
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case Connector.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}
    {:ok, 0} -> {:ok, :duplicate}
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}
    other -> other
  end
end

Beat two — the script body. @enqueue is the protocol: admit by kind, refuse a duplicate, write the row, insert the pending entry — one atomic step. KEYS[1] is the job row, KEYS[2] the pending set; ARGV[1] is the branded id, ARGV[2] the payload.

-- the @enqueue script (L2) — one atomic transition, every key declared
-- The branded-id gate: the id must be JOB-namespaced, refused at the wire.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
-- Idempotent: a row that already exists is a no-op (a duplicate enqueue).
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
-- The three-field row: state, attempts, payload — the shape every speaker reads.
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- The pending set: the id is the member, so byte order is mint order.
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

What the excerpt shows is the boundary: the KEYS/ARGV contract, the branded-id gate, the three named fields. None of it is language-specific. The executor that runs it — load-once, run-by-SHA — is L3, above the line, and each runtime writes that its own way.

Because the keys and the Lua are the whole agreement, any runtime that speaks them is a peer — so EchoMQ is supported in, and ported to, other runtimes; the depth of this course is the canonical one, Elixir.

Draw the line between L2 and L3, and a Valkey convention becomes a protocol three pillars share and any runtime can speak. Below it: one key format, one set of declared-key scripts. Above it: your language.

Why L1 and L2 are fixed — change a field, who breaks?

The fields state, attempts, payload are L1, and so is the key prefix. They are the contract, not an implementation detail. Rename payload to data in one runtime's executor and that runtime's @enqueue writes a field nobody else reads — every other speaker reads payload and finds nothing. The data is not lost; it is invisible across the wire. The same is true of the key: change emq: and your jobs land in a keyspace no other speaker looks in. The L4 verb is the opposite — it is above the line, so renaming it breaks nothing. Make one change below the line and read who can still find the row.

Change one thing · who can still read the job?

below the line

above the line

the pattern (Redis Patterns Applied)

A Valkey convention becomes a protocol when the agreement is pushed below the line — the keys plus the atomic scripts — and the runtime is left above it. Redis Patterns Applied teaches that move; this is the system that applies it.

the implementation (echo_mq)

EchoMQ.Keyspace.queue_key/2 fixes the key emq:{q}:<type> with the {q} hashtag; the @enqueue script fixes the transition with every key declared in KEYS[]. L1 and L2, shared and fixed.

Below the line is the contract; above it is the code. A field rename below the line makes the row invisible to every other speaker; an L4 rename above it changes nothing on the wire.

Recap — the line, the key, the script

The protocol below the line is the keys and the Lua. EchoMQ.Keyspace.queue_key/2 builds emq:{q}:<type>, where the {q} hashtag pins one queue to one slot so one atomic script can touch its sets and rows together. EchoMQ.Jobs's @enqueue script declares every key it touches in KEYS[], gates the branded id, and writes the three-field row and the pending entry atomically. That shared substrate — L1 and L2 — is fixed; the executor (L3) and the API (L4) are the runtime's own. The next dive reads where EchoMQ lands: the door from Redis Patterns Applied and the BCS family.

Next in the Overview: The door & the BCS family — the bidirectional Redis-Patterns door, and what owning the queue, the bus, and the cache means for the systems built on them.

References

Where the layer model is grounded, and where the keyspace and scripting layers are documented.

Sources

Valkey — Documentation — the BSD-licensed, foundation-governed store EchoMQ is backed by; the substrate of record.
Valkey — EVALSHA — the load-once, run-by-SHA dispatch the L3 executor uses to run an L2 script.
Redis — Keyspace & hash tags — cluster key routing by the hashtag inside {...}, the L1 mechanism the {q} brace uses.
DragonflyDB — Server flags — --lock_on_hashtags: the thread-per-shard placement the declared-key, per-queue-hashtag keyspace is built for.

Related in this course

Overview — the chapter this dive belongs to; the three pillars over one wire.
The Protocol — the owned keyspace, the record hash, and the Lua layer in depth (the next chapter the line opens onto).
redis-patterns · Patterns become protocol — the near side of the door: the move from convention to protocol.
The Branded Component System — the architecture EchoMQ is the bus of; the branded id the @enqueue gate enforces is its identity contract.
Course home — the full six-chapter map.