jonnifyechomq
EchoMQ / E2 · Core / The three runtimes / Cross-runtime interop

E2.04.3 · The three runtimes · dive 3

Cross-runtime interop

A job written by one runtime is picked up and finished by another — and the v2 fork makes interop the EchoMQ fleet's own.

v1 interop works today because the wire is shared: one job hash at emq:{queue}:{jobId}, the same compressed fields — including atm — and the same transition scripts. The Go port and the Elixir reference write and read those fields identically. The honest seam: Node.js is the BullMQ origin, and EchoMQ has no first-party Node runtime today. The v2 trajectory: emq.1 ships EchoMQ 2.0 — the emq:{q}:… keyspace, every Lua key declared, the transparent {q} hashtag, echomq:2.0.0 with a two-way fence. Step the round-trip on the right.

A job round-trip · Go writes, Elixir reads, atm survives

1 · Go enqueues, writes atm 2 · Elixir worker reads atm 3 · processing increments atm emq:{orders}:{jobId} atm = 0

v1 interop today — one job hash, shared scripts

A standard job lives in one Redis hash at emq:{queue}:{jobId} with the same compressed field names in every runtime — including atm (attempts made). The Go port and the Elixir reference write and read that field identically:

// Go — write atm on the hash (queue_impl.go ~102), read it back (script_runner.go ~160)
"atm": job.AttemptsMade
data["atm"]
# Elixir — Job.from_redis/4 reads the same compressed fields: atm/ats/stc/deid/defa/nrjid
EchoMQ.Job.from_redis(raw_fields, queue, prefix, conn)

So a job enqueued by Go and fetched by an Elixir worker — or the reverse — agrees on every field, because the field names are L1 and the transition scripts are L2. The honest seam: Node.js is the BullMQ reference implementation, and EchoMQ has no first-party Node runtime today. v1 interop in practice is Elixir ↔ Go ↔ upstream-BullMQ-Node, all on the shared emq: wire.

v1 interop works today because the field names and scripts are shared. A job round-trips between any two runtimes that speak the emq: wire — Elixir, Go, and upstream Node.

v1 vs v2 — the keyspace comparator

The shared wire above is the v1 line (frozen at 1.3.0). emq.1 ships EchoMQ 2.0, which leaves the BullMQ wire deliberately, once, and versioned. Pick a key type to compare the v1 key against what emq.1 ships as its v2 form — the emq: prefix, and the per-queue {q} hashtag the core places transparently. The v2 side is always what emq.1 ships, not shipped code.

emq:… (v1 line) vs emq:{q}:… (emq.1 ships)

v1 line (frozen at 1.3.0) — no hashtag emq:orders:wait emq.1 ships — the {orders} hashtag placed by the core emq:{orders}:wait

The same key takes two forms: emq:orders:wait on the v1 line, and the emq:{orders}:wait that emq.1 ships — the emq: prefix and the core-placed {q} hashtag.

The fleet — interop becomes EchoMQ's own

Once the wire is EchoMQ's own, interop is the fleet's own too:

  • Stock BullMQ clients cannot speak v2 — by design. A v2 deployment never reads or writes emq:*; that is the honest fleet seam.
  • emq.1 ships a fleet-interop probe — a job enqueued by one first-party runtime is fetched and completed by another, both directions, the harness runtime-pluggable. Honest status until then: Elixir is the only v2 speaker; the Go port must implement 2.0 to join.
  • A first-party echomq-node is proposed — a Node.js runtime with the Dragonfly target and the same transparent {key} convention. It is proposed; it does not exist.

The v2 fork makes interop the EchoMQ fleet's own: stock BullMQ clients out by design, the Go port to be ported, the proposed echomq-node ahead — with emq.1's fleet-interop probe tracking the honest status.

Dragonfly — the v2 first-class substrate

The payoff of every key declared and every queue hashtagged is DragonflyDB-native multithreading. Dragonfly is thread-per-shard: a key's {hashtag} decides its owning thread. With v2, --lock_on_hashtags locks precisely — one queue per thread, distinct queues across cores — and the deployment never needs --default_lua_flags=allow-undeclared-keys, the whole-store escape hatch that v1's undeclared keys would otherwise force. The two flags are the v2 substrate's posture: require --lock_on_hashtags, never set the allow-undeclared-keys flag.

--lock_on_hashtags
locks at the {hashtag} level — one queue per thread under v2.
allow-undeclared-keys
the whole-store escape hatch v1 would force; emq.1 ships v2 so it is never needed.

The Golden Rule — protocol, then its runtimes

Interop is a property of the wire, not of any one runtime. One job hash, the same field names (atm …, L1), the same transition scripts (L2) — and a job round-trips between any two runtimes that share them. Each runtime reads and writes those keys its own way (L3/L4): Elixir with OTP processes, the Go port with goroutines, and (in v2) the proposed echomq-node with its event loop.

the protocol (immutable L1/L2)

One job hash, the same field names (atm …), the same transition scripts — shared, so a job round-trips between any two runtimes on the same wire.

its three runtimes (variable L3/L4)

Elixir reads and writes the fields with OTP processes, the Go port with goroutines, and (in v2) the proposed echomq-node with its event loop — each a different executor over the same declared keys.

v1 interop works today because the field names and scripts are shared. The v2 fork makes interop the EchoMQ fleet's own — stock BullMQ clients out by design, the Go port to be ported, echomq-node proposed — with Dragonfly the first-class multithreaded substrate the break unlocks.

This dive closes the module. Back to E2.04 · the three runtimes, or up to E2 · the core for the chapter.

The 2.0 fork — EchoMQ leaves the BullMQ wire

This dive is the fork in full. The shared emq: wire is the v1 line (frozen at 1.3.0); emq.1 ships EchoMQ 2.0 — the emq: keyspace, every Lua key declared in KEYS[], the transparent {q} hashtag, echomq:2.0.0 with a two-way boot fence — and the migration is drain-and-switch plus an at-rest key tool. Interop turns inward to the fleet (stock BullMQ clients out by design, the Go port to be ported, the proposed echomq-node), and Dragonfly is the first-class multithreaded substrate the break unlocks.

References

The shared wire and job-hash field set, and the Dragonfly substrate the v2 fork targets.

Sources

Related in this course