The versioned break

A break is only safe if it is governed. EchoMQ 2.0 breaks once, stamps a new version, fences every mixed contact, and ships an explicit migration.

The owned keyspace says what changes on the wire. This dive says how the change is governed: once, versioned, and fenced. emq.1 ships meta.version = echomq:2.0.0 (the v1 line, recording bullmq:5.65.1 today, freezes at 1.3.0), a two-way typed boot fence that refuses any cross-version contact, and an explicit v1→v2 migration path. The v2 script set is dispatched the proven way the shipped EchoMQ.Scripts.execute_raw/4 dispatches today. The rung is emq.1, drafted, taught from its spec.

Movement II · Extension · tracks emq.1 · drafted (the 2.0 break) ← redis-patterns R0 ← redis-patterns R2

Versioned and fenced The dispatch the v2 set rides The branded ids References

EVALSHA → EVAL fallback · the v2 set, dispatched the proven way

Versioned and fenced

The owned keyspace states what changes on the wire. This dive states how the change is governed — once, versioned, and fenced — so a v2 worker and a v1 keyspace never touch silently. emq.1 ships three pieces of that governance: the version bump, a two-way boot fence, and an explicit migration. The dispatch itself does not change.

Today the v1 line records its wire version in meta.version: EchoMQ.Version.full_version/0 returns "bullmq:#{@bullmq_version}" (version.ex:54), value bullmq:5.65.1. emq.1 ships EchoMQ.Version recording echomq:2.0.0 (D3), and a two-way typed boot fence: a v2 worker against a bullmq:* keyspace (or emq:* keys) refuses with a typed error, and a v1 worker against an echomq:2.0.0 keyspace refuses too. No silent cross-version read or write, ever (INV1). The v1 line freezes at 1.3.0 — a maintenance branch, not deleted code.

meta.version (emq.1 ships): D3: echomq:2.0.0 replaces the v1 line's bullmq:5.65.1 (EchoMQ.Version, version.ex:54). The v1 line freezes at 1.3.0.
the two-way fence (emq.1 ships): D3: a v2 worker against a v1 (emq:*) keyspace refuses with a typed error at boot, and a v1 worker against a v2 keyspace likewise — both directions fenced (INV1).
EchoMQ.Ext.ID (emq.1 ships): D4: integer Snowflakes (epoch 2024-01-01), a decimal-string wire form, base62 branding under the BAT namespace at the API edge (INV5).

The rung is drafted. lib/echomq/ext/ and the v2 script set do not exist yet, so every v2 surface here is taught from specs/emq/emq.1.md, written "emq.1 ships …" — never asserted as shipped. EchoMQ.Scripts.execute_raw/4 and EchoMQ.Version (bullmq:5.65.1) are real and shipped — the present tense is the v1 line's.

The dispatch the v2 set rides

The shipped core loads its 50 Lua scripts by one rule, and EchoMQ.Scripts.execute_raw/4 is that rule in code (@spec execute_raw(atom(), String.t(), [String.t()], [any()])). On every call it computes the script's SHA, then issues EVALSHA <sha> <numkeys> KEYS… ARGV… over the pooled EchoMQ.RedisConnection (a use Supervisor, NimblePool-backed pool with command/3, command!/3, and pipeline/3). If the script is cached, one round trip runs it. If Redis answers NOSCRIPT, execute_raw/4 falls back to EVAL <source> <numkeys> …, which runs the script and caches it — so the next EVALSHA is a hit. Two commands at most, a warm cache after the first miss. That is the R0 foundation: load once, dispatch by SHA. This dispatch is shipped today.

emq.1 ships the v2 script set dispatched the same proven way — EVALSHA first, EVAL on NOSCRIPT, on the same pooled connection. The break is not in the dispatch; it is in the scripts. Every v2 script declares every operand key in KEYS[] (D2), so the v1 pattern local jobKey = keyPrefix .. jobId (moveToActive-11.lua:148, from an ARGV prefix) is gone. The hero figure steps the real two-command sequence over a fixed v2 call.

# EchoMQ.Scripts.execute_raw/4 — the shipped dispatch (priv/scripts), EVALSHA-first.
# @spec execute_raw(atom(), String.t(), [String.t()], [any()]) :: script_result()
sha = sha1(script)
case RedisConnection.command(conn, ["EVALSHA", sha, num_keys | keys ++ encoded_args]) do
  {:ok, result} -> {:ok, decode_result(result)}     # cache HIT
  {:error, %Redix.Error{message: "NOSCRIPT" <> _}} ->
    RedisConnection.command(conn, ["EVAL", script, num_keys | keys ++ encoded_args]) # runs + caches
end

# emq.1 ships the v2 set dispatched the SAME way — every key declared in KEYS[] (D2).
# the dispatch is unchanged; what changes is meta.version + the fence:
  "bullmq:5.65.1"  # v1 line, frozen at 1.3.0 (EchoMQ.Version, version.ex:54)
  "echomq:2.0.0"   # emq.1 ships — a v2 worker fences a emq:* keyspace, and vice versa (INV1)

The dispatch is the v1 line's, proven and unchanged: load once, EVALSHA first, EVAL on a miss. The break is in the version, the fence, and the declared-keys scripts.

The two-way fence, and the migration

Once a queue is initialised, its meta.version is on record. emq.1 ships a boot fence on both directions. A v2 worker reads the queue's meta.version: if it sees bullmq:* (or emq:* keys), it refuses to boot with a typed error rather than write emq:{q}:* keys into a v1 keyspace. A v1 worker, conversely, refuses an echomq:2.0.0 keyspace. A mixed deployment fails fast at boot, not at runtime (INV1). The matrix below resolves each (worker × keyspace) pairing.

A v1→v2 cutover is not automatic, and emq.1 ships it as an explicit path (D7). The drain-and-switch runbook: stop producers; drain the v1 queues with v1 workers; start v2. An at-rest key-migration tool rewrites jobs not in flight — emq:q:* → emq:{q}:*, the job field set preserved modulo the versioned meta. The runbook names what cannot migrate live: in-flight active jobs hold v1 locks and must drain on v1 workers first. The migration is a procedure, not a silent swap — that honesty is the point.

The two-way fence · worker × keyspace

A break is only safe if it is governed. The fence makes any cross-version contact a typed boot error, and the migration makes the cutover a named procedure — never a silent swap.

The branded ids

emq.1 ships EchoMQ.Ext.ID to mint identifiers for extension state (emq.1-D4). The id is an integer Snowflake: epoch 2024-01-01, layout ts(41)<<22 | node(10)<<12 | seq(12). Decoding is shift-and-mask — ts = id >> 22, node = (id >> 12) & 0x3FF, seq = id & 0xFFF. Over the verified pair 274557032793636864 that yields ts = 65459497641, node = 0, seq = 0.

The wire form is a decimal string; the branded form is the 11-char base62 of the integer under the BAT namespace, minted only at the API edge. The codec is the same base62 Snowflake codec this page's build stamp uses, so the check is direct: 274557032793636864 base62-encodes to 0KHTOWnGLuC, and the spec's known pair TSK0KHTOWnGLuC ⇄ 274557032793636864 shares that exact 11-char body — BAT0KHTOWnGLuC and TSK0KHTOWnGLuC differ only at the three-letter namespace edge. The discipline (INV5): ids cross every boundary as decimal integer strings, branding exists only at the edge, and Ext.ID.decode(Ext.ID.brand(x)) == x holds by property. The figure round-trips a small fixed set to show that identity.

Decode the Snowflake · brand & round-trip

the pattern · ← redis-patterns R0 (foundations)

Scripts load once and dispatch by SHA over a pooled connection — the keyspace and the scripts are the protocol, run on a warm cache after the first miss.

the implementation · ⇄ emq.1 (drafted · the 2.0 break)

emq.1 ships the v2 script set dispatched the same EVALSHA→EVAL way the shipped EchoMQ.Scripts.execute_raw/4 dispatches today — the dispatch unchanged; the break in the version (echomq:2.0.0), the two-way fence, and every key declared.

The atomicity floor carries through (INV4): every v2 script is pure over KEYS/ARGV, time passed as an ARGV millisecond — so every transition stays replayable, exactly as the v1 scripts are.

The 2.0 fork — EchoMQ leaves the BullMQ wire

bullmq:5.65.1 is the v1 line's meta.version, frozen at 1.3.0 (E2 · the core teaches it). emq.1 ships the break: meta.version becomes echomq:2.0.0, a two-way typed boot fence refuses any cross-version contact, and an explicit drain-and-switch runbook plus an at-rest key-migration tool carry a deployment over — once, versioned, never silently.

Recap — the break, governed

emq.1 ships the governance of the break. EchoMQ.Version records echomq:2.0.0 (the v1 line, recording bullmq:5.65.1 today, freezes at 1.3.0). A two-way typed boot fence refuses any cross-version contact — a v2 worker against a v1 keyspace and a v1 worker against a v2 one both fail fast at boot (INV1). An explicit v1→v2 migration path — drain-and-switch plus an at-rest key-migration tool — carries a deployment over, naming honestly that in-flight active jobs cannot migrate live. The v2 script set is dispatched the proven way EchoMQ.Scripts.execute_raw/4 dispatches the v1 scripts today; the difference is that every v2 script declares every key (D2). And EchoMQ.Ext.ID mints integer Snowflakes branded BAT… at the edge, decimal across boundaries (INV5).

Next in E3: Conformance, not compatibility — how emq.1 turns the break into a standing v2 conformance probe and a fleet-interop probe, so a regression is caught rather than assumed.

References

Where this dive is grounded: the BullMQ wire the v1 line speaks and the break leaves, the script-dispatch commands the v2 set rides, and the chapter's own pages.

Sources

BullMQ — Documentation — the reference implementation whose wire the v1 line speaks (bullmq:5.65.1) and the 2.0 break leaves behind.
Redis — EVALSHA — the run-by-SHA dispatch the v2 set rides, exactly as execute_raw/4 does today.
Redis — SCRIPT LOAD — how a script is cached at init for SHA dispatch.
Redis — Documentation — the scripting and data-structure reference under the dispatch.

Related in this course

E3 · EchoMQ 2.0 — the chapter this dive sits in.
E2 · The core — the v1 line the fork freezes.
redis-patterns R0 · Overview — foundations: load-once, SHA-dispatched scripts.
redis-patterns R2 · Coordination — atomicity, the floor the v2 scripts keep.