jonnifyechomq

The Lua layer · dive 02

Declared keys

The host builds every key; the script touches only KEYS[]. Values ride ARGV. That is the law.

Every key a script touches is declared in KEYS[]; the script constructs none from data. EchoMQ.Script's own contract states it — every key a script touches is declared in KEYS; ARGV carries values only. It is the discipline that keeps atomicity sound on a cluster, keeps the keyspace the single owner of where data lives, and keeps the wire portable to a thread-per-shard engine. Classify a key on the right, then read the two beats below.

The host builds, the script declares Why it is the law Co-location, on a slot References

Declared or constructed? · classify a key

@enqueue (Lua) host → KEYS[1] host → ARGV[1] (value) 'emq:'..ARGV (in-script)

The host builds, the script declares

Two beats again: the handle that builds the keys, and the body that touches only what it was handed. The difference this dive draws is which things cross as keys and which cross as values.

Beat one — the host builds the keys. EchoMQ.Jobs.enqueue/4 builds both keys through EchoMQ.Keyspace — the single owner of where data lives — then hands them to the script as KEYS. The values go in ARGV. The script never sees the queue name, the braces, or the slot; it sees two opaque keys and two opaque values.

# echo_mq — EchoMQ.Jobs
# The host builds KEYS through the keyspace, the single owner of where data
# lives. The script receives [row, pending] declared and [id, payload] as
# values. The script does not know the queue name, the braces, or the slot —
# only the two keys it was handed.
def enqueue(conn, queue, job_id, payload) when is_binary(job_id) and is_binary(payload) do
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]
  Connector.eval(conn, @enqueue, keys, [job_id, payload])
  # ... verdict mapping omitted; see scripts-are-the-protocol
end

Beat two — the script touches only declared keys. @enqueue references KEYS[1] and KEYS[2] and nothing else. It never builds a key by concatenating ARGV. The branded id rides ARGV[1] as a value — the member written into the set, never spliced into a key name.

-- the @enqueue script — KEYS[1] = the job row, KEYS[2] = the pending set
-- Both keys arrive declared; the script constructs neither. ARGV[1] (the id)
-- is a VALUE — the member written into KEYS[2], never concatenated into a key.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

The id appears inside the script only as the third argument to ZADD — a member, a value. It is never used to build the row key; that key was built host-side and arrived as KEYS[1]. The rule for where the job lives stays in EchoMQ.Keyspace, where it belongs.

Keys are handed in; values ride ARGV. A script that builds a key from data has moved the address rule out of the keyspace and broken the law.

Why declared keys is the law — three reasons

The discipline is not a style choice. Three things depend on it.

One — atomicity needs co-location. A Valkey cluster runs one script's keys on one node only if they hash to one slot. The per-queue hashtag {q} puts every key of a queue on one slot; declaring those keys is how the engine verifies the script stays on its slot. A key constructed in-script is not on the declared set, so the engine cannot guarantee it is co-resident.

Two — the keyspace stays the single owner of address. If a script built a key from data, the rule for where a job lives would be split between EchoMQ.Keyspace and the script body. Declaring keys keeps that rule in one place. There is exactly one answer to "where does this job's row live", and it is the keyspace's.

Three — thread-per-shard placement. A multithreaded engine like Dragonfly places a script on a thread by its declared key set (--lock_on_hashtags). A key constructed inside the script is invisible to that placement — so an undeclared key is not portable to a thread-per-shard engine. Conformance is phrased against Valkey; the declared-key discipline is what makes the multithreaded placement reachable.

Co-location, on a slot

One enqueue touches the row and the pending set. Both are declared, both carry the {q} hashtag, so both hash to one slot and the script runs on one node — atomic. Add an undeclared key (built in the script) and the guarantee is gone: the key is not in the declared KEYS set, so the engine cannot place the script where that key lives. Compare the two below.

Declared keys vs an undeclared one · pick a script

emq:{q}:job:<id> KEYS[1] · slot S emq:{q}:pending KEYS[2] · slot S —
the pattern (Redis Patterns Applied)

Hash-tag co-location and one-slot atomic Lua: keep the keys a script touches on one slot so one script can touch them atomically. Hash-tag co-location and Coordination teach it.

the implementation (echo_mq)

@enqueue declares KEYS[1] and KEYS[2], both built by EchoMQ.Keyspace on the {q} slot; the script builds no key from data, so the engine can place it.

Declared keys is the law because it is what atomicity, single ownership of address, and thread-per-shard placement all stand on.

Recap — declared keys, three reasons

The host builds every key; the script touches only KEYS[]; values ride ARGV. It keeps atomicity sound on a cluster, keeps the keyspace the single owner of address, and keeps the wire portable to a thread-per-shard engine. The next dive reads how the declared-keys script is dispatched — loaded once and run by its SHA.

Next in the Lua layer: EVALSHA dispatch — the SHA precomputed at definition, the script run by digest, the source loaded once on a NOSCRIPT miss.

References

Where the declared-keys discipline is grounded, and the engine behaviour it serves.

Sources

Related in this course