The Lua layer · dive 01

Scripts are the protocol

One transition, one script, atomic. The enqueue protocol is the enqueue script — read it in two beats.

A state change in EchoMQ is not a sequence of client commands — it is one server-side Lua script that runs atomically. The enqueue transition is the @enqueue script: it admits the job by kind, refuses a duplicate, writes the row, and inserts the pending entry, in one indivisible step. No client sees the row half-written; no concurrent enqueue writes it twice. Step the four moves on the right, then read the two beats below.

The two beats Run @enqueue Pattern & implementation References

The @enqueue transition · step the moves

The enqueue transition, in two beats

Read a transition the way the protocol is built: the handle (the Elixir verb that builds the keys and runs the script) and the body (the Lua that performs the transition). The handle is above the line; the body is the protocol.

Beat one — the named handle. EchoMQ.Jobs.enqueue/4 builds the two keys the transition touches — the job row and the pending set — and runs the @enqueue script through the connector. The keys are built host-side by EchoMQ.Keyspace; the values ride ARGV. The script's integer return becomes a verdict.

# echo_mq — EchoMQ.Jobs
# enqueue/4 is the handle. It builds the two keys @enqueue will touch — the
# job row and the pending set — and hands them in declared. The connector
# runs @enqueue EVALSHA-first. KEYS = [row, pending]; the values (id, payload)
# ride ARGV. The script's integer return is mapped to a verdict.
def enqueue(conn, queue, job_id, payload) when is_binary(job_id) and is_binary(payload) do
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case Connector.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}      # the row was new — written and made pending
    {:ok, 0} -> {:ok, :duplicate}     # a row already existed — idempotent no-op
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}  # id not JOB-namespaced
    other -> other
  end
end

Beat two — the script body. @enqueue is the protocol: the four moves the stepper above walks, in one atomic transition. KEYS[1] is the job row, KEYS[2] the pending set; ARGV[1] is the branded id, ARGV[2] the payload.

-- the @enqueue script — one atomic transition, every key declared
-- The branded-id gate: the id must be JOB-namespaced (first three bytes 'JOB'),
-- refused at the wire. The gate runs in Lua, so no malformed id can be admitted
-- by any speaker — the rule lives below the language line.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
-- Idempotency: a row that already exists is a no-op. Two enqueues of the same
-- id leave exactly one row and one pending entry — the second returns 0.
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
-- The three-field row: state, attempts, payload — the shape every speaker reads.
-- attempts starts at '0' (the fencing token); state starts at 'pending'.
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
-- The pending set: the id itself is the member at score 0, so byte order is
-- mint order and the queue carries no second index.
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

Every key the script touches arrives declared in KEYS; every value rides ARGV. The four moves are one transition: the row and the pending entry appear together or not at all. None of it is language-specific — the handle (beat one) is the runtime's; the transition (beat two) is the wire's.

A transition is one script. Read it in two beats — the handle that builds the keys, then the body that is the protocol — and the wire stops being a list of commands and becomes a set of scripts.

Run @enqueue — the three verdicts

The script returns one of three things, and the handle maps each to a verdict. Enqueue a job, enqueue it again, and try a malformed id — read who is admitted and why. The model below is the real script's logic over a fixed in-memory keyspace: EXISTS the row, gate the id's first three bytes, return 1 / 0 / an EMQKIND error.

Enqueue against a fresh queue · run the script

The pending set and the row move together: :enqueued writes both, :duplicate writes neither, and a bad kind never reaches the row. That is the atomicity of one script — there is no state where the row exists but the pending entry does not.

Pattern & implementation

the pattern (Redis Patterns Applied)

An atomic multi-step update, expressed as one server-side script, becomes a protocol when the agreement is pushed below the language line. Patterns become protocol teaches the move; atomic updates is the near side of the door.

the implementation (echo_mq)

EchoMQ.Jobs's @enqueue runs the four-move transition atomically — gate, idempotency, row, pending — and enqueue/4 is the handle that builds the keys and maps the verdict.

The pattern says: push the transition below the line as one atomic script. The implementation is @enqueue — the enqueue protocol, in one indivisible step.

Recap — the transition is the script

The enqueue transition is one atomic Lua script: gate the branded id, refuse a duplicate, write the three-field row, insert the pending entry. Read it in two beats — enqueue/4 the handle, @enqueue the transition. The next dive reads the law the script obeys: every key declared in KEYS[], none built from data.

Next in the Lua layer: Declared keys — the host builds the keys, the script touches only KEYS[], and why that is what makes the wire portable.

References

Where the atomic-script transition is grounded, and the commands the body uses.

Sources

Redis — EVAL — atomic server-side scripting; a script runs to completion with no interleaving.
Valkey — EVALSHA — the load-once dispatch the handle runs the script with.
Valkey — HSET — the hash write the three-field row is built from.
Valkey — ZADD — the pending-set insertion the transition ends with.

Related in this course

The Lua layer — the module this dive belongs to.
Declared keys — the law the script obeys.
The record hash — the three-field row the script writes.
redis-patterns · Atomic updates — the atomic-update pattern, the near side of the door.
redis-patterns · Patterns become protocol — convention becoming protocol.