E2.04.3 · The three runtimes · dive 3
Cross-runtime interop
A job written by one runtime is picked up and finished by another — and the v2 fork makes interop the EchoMQ fleet's own.
v1 interop works today because the wire is shared: one job hash at
emq:{queue}:{jobId}, the same compressed fields — including
atm — and the same transition scripts. The Go port and the Elixir reference write and
read those fields identically. The honest seam: Node.js is the BullMQ origin, and EchoMQ has no
first-party Node runtime today. The v2 trajectory: emq.1 ships EchoMQ 2.0 — the
emq:{q}:… keyspace, every Lua key declared, the transparent
{q} hashtag, echomq:2.0.0 with a two-way fence. Step the
round-trip on the right.
Movement I · Core (as-built) ← redis-patterns R3
A job round-trip · Go writes, Elixir reads, atm survives
v1 interop today — one job hash, shared scripts
A standard job lives in one Redis hash at emq:{queue}:{jobId} with the same
compressed field names in every runtime — including atm (attempts made). The Go port
and the Elixir reference write and read that field identically:
// Go — write atm on the hash (queue_impl.go ~102), read it back (script_runner.go ~160) "atm": job.AttemptsMade data["atm"]
# Elixir — Job.from_redis/4 reads the same compressed fields: atm/ats/stc/deid/defa/nrjid EchoMQ.Job.from_redis(raw_fields, queue, prefix, conn)
So a job enqueued by Go and fetched by an Elixir worker — or the reverse — agrees on every field, because the
field names are L1 and the transition scripts are L2. The honest seam:
Node.js is the BullMQ reference implementation, and EchoMQ has no first-party Node runtime today.
v1 interop in practice is Elixir ↔ Go ↔ upstream-BullMQ-Node, all on the shared
emq: wire.
v1 interop works today because the field names and scripts are shared. A job round-trips between any
two runtimes that speak the emq: wire — Elixir, Go, and upstream Node.
v1 vs v2 — the keyspace comparator
The shared wire above is the v1 line (frozen at 1.3.0). emq.1 ships
EchoMQ 2.0, which leaves the BullMQ wire deliberately, once, and versioned. Pick a key type to compare the v1
key against what emq.1 ships as its v2 form — the emq: prefix, and the
per-queue {q} hashtag the core places transparently. The v2 side is always what emq.1
ships, not shipped code.
emq:… (v1 line) vs emq:{q}:… (emq.1 ships)
The same key takes two forms: emq:orders:wait on the v1 line, and the
emq:{orders}:wait that emq.1 ships — the
emq: prefix and the core-placed {q} hashtag.
The fleet — interop becomes EchoMQ's own
Once the wire is EchoMQ's own, interop is the fleet's own too:
- Stock BullMQ clients cannot speak v2 — by design. A v2 deployment never reads or writes
emq:*; that is the honest fleet seam. - emq.1 ships a fleet-interop probe — a job enqueued by one first-party runtime is fetched and completed by another, both directions, the harness runtime-pluggable. Honest status until then: Elixir is the only v2 speaker; the Go port must implement 2.0 to join.
- A first-party echomq-node is proposed — a Node.js runtime with the Dragonfly target and the
same transparent
{key}convention. It is proposed; it does not exist.
The v2 fork makes interop the EchoMQ fleet's own: stock BullMQ clients out by design, the Go port to be ported, the proposed echomq-node ahead — with emq.1's fleet-interop probe tracking the honest status.
Dragonfly — the v2 first-class substrate
The payoff of every key declared and every queue hashtagged is DragonflyDB-native
multithreading. Dragonfly is thread-per-shard: a key's {hashtag} decides its
owning thread. With v2, --lock_on_hashtags locks precisely — one queue per thread,
distinct queues across cores — and the deployment never needs
--default_lua_flags=allow-undeclared-keys, the whole-store escape hatch that v1's
undeclared keys would otherwise force. The two flags are the v2 substrate's posture: require
--lock_on_hashtags, never set the allow-undeclared-keys flag.
- --lock_on_hashtags
- locks at the
{hashtag}level — one queue per thread under v2. - allow-undeclared-keys
- the whole-store escape hatch v1 would force; emq.1 ships v2 so it is never needed.
The Golden Rule — protocol, then its runtimes
Interop is a property of the wire, not of any one runtime. One job hash, the same field names
(atm …, L1), the same transition scripts (L2) — and
a job round-trips between any two runtimes that share them. Each runtime reads and writes those keys its own way
(L3/L4): Elixir with OTP processes, the Go port with goroutines, and (in v2) the proposed
echomq-node with its event loop.
One job hash, the same field names (atm …), the same transition scripts — shared, so
a job round-trips between any two runtimes on the same wire.
Elixir reads and writes the fields with OTP processes, the Go port with goroutines, and (in v2) the proposed echomq-node with its event loop — each a different executor over the same declared keys.
v1 interop works today because the field names and scripts are shared. The v2 fork makes interop the EchoMQ fleet's own — stock BullMQ clients out by design, the Go port to be ported, echomq-node proposed — with Dragonfly the first-class multithreaded substrate the break unlocks.
This dive closes the module. Back to E2.04 · the three runtimes, or up to E2 · the core for the chapter.
The 2.0 fork — EchoMQ leaves the BullMQ wire
This dive is the fork in full. The shared emq: wire is the v1 line (frozen at
1.3.0); emq.1 ships EchoMQ 2.0 — the emq: keyspace, every Lua
key declared in KEYS[], the transparent {q} hashtag,
echomq:2.0.0 with a two-way boot fence — and the migration is drain-and-switch plus an
at-rest key tool. Interop turns inward to the fleet (stock BullMQ clients out by design, the Go port to be
ported, the proposed echomq-node), and Dragonfly is the first-class multithreaded substrate the break unlocks.
References
The shared wire and job-hash field set, and the Dragonfly substrate the v2 fork targets.
Sources
- BullMQ — Documentation — the shared
emq:wire and the job hash field set both runtimes use. - Redis — EVALSHA — the cached-script dispatch the shared scripts run through.
- DragonflyDB — Running BullMQ on Dragonfly — the thread-per-shard substrate the v2 fork targets first-class.
- DragonflyDB — Server flags —
--lock_on_hashtagsand theallow-undeclared-keysescape hatch the break avoids. - Redis — Documentation — Redis Cluster and hash tags underneath both keyspaces.
Related in this course
- E2.04 · The three runtimes & interop — the module hub.
- E2.04.1 · The Elixir architecture — the reference runtime that speaks v2 first.
- E2.04.2 · The Go gaps — the Go port's v2 porting work-list.
- E1.02 · The field-name bug — the
attemptsMade/atmfield round-trip in depth. - redis-patterns R3 · Reliable queues — the pattern the shared wire applies.