jonnifyechomq
EchoMQ / E3 · EchoMQ 2.0 / Conformance, not compatibility

E3 · EchoMQ 2.0 · Movement II · orientation

Conformance, not compatibility

EchoMQ 2.0 broke from BullMQ on purpose. So it does not check that it stayed compatible — it checks that it conforms to its own protocol.

The v1 line was measured against BullMQ: did the wire stay the same? EchoMQ 2.0 retires that question. emq.1 ships two new probes that measure the break instead — a v2 conformance probe and a fleet-interop probe. The rung is drafted: the v2 script set and lib/echomq/ext/ do not exist on disk yet, so this dive teaches from specs/emq/emq.1.md ("emq.1 ships …") and is re-grounded in the real harnesses when it ships. The v1 flaw the break resolves is real and present. This is the orientation dive; the chapter's modules sit on the chapter landing.

Movement II · tracks emq.1 · drafted (the 2.0 break) ← redis-patterns R0 ← redis-patterns R2

The conformance probe The fleet-interop probe The v1 flaw it resolves References

The v2 conformance probe · queue payments

v2 capture · emq:{payments}:wait → :active + :<jobId>:lock meta.version = echomq:2.0.0 · emq:* reads/writes = 0

    Probe one — the v2 conformance probe

    The v1 line asked "did the wire stay compatible with BullMQ?" EchoMQ 2.0 broke from BullMQ on purpose, so the question changes. emq.1 ships the v2 conformance probe (emq.1-D5): an add → process round-trip on a v2 queue asserts the deployment conforms to the 2.0 protocol — it does not measure against the wire it left.

    The probe checks five things against the captured transition. The produced key set is emq:{q}:*-only — no emq: prefix anywhere. Every key of the transition shares one hashtag (the queue's {payments}), so it is single-slot. There are zero emq:* reads or writes. The scripts ran with every key declared — the suite traces every operand key to KEYS[], no escape hatch. And meta.version reads echomq:2.0.0.

    The checklist runner in the figure above ticks each assertion over a fixed v2 capture — step through them one at a time, or assert all five at once.

    emq:{q}:*-only
    The produced key set is the owned keyspace; no emq: prefix appears (INV1).
    declared keys, no escape hatch
    Every operand key is in KEYS[]; the suite runs on Valkey, the conformance truth, and asserts the declaration discipline directly (INV2).
    one hashtag · version
    A single hashtag per queue (single-slot, atomic), and meta.version = echomq:2.0.0.

    Conformance is not a promise against an old wire. emq.1 ships a probe that asserts the deployment conforms to the 2.0 protocol — emq:{q}:*-only, one hashtag, zero emq:*, every key declared.

    Probe two — the fleet-interop probe

    Conformance on one runtime is half the proof. The other half is that a job crosses the wire between first-party runtimes, both directions. emq.1 ships the fleet-interop probe (emq.1-D6): a job enqueued by one v2 runtime is fetched and completed by another, and the reverse round-trips too. The shipped EchoMQ.Queue.add/4 is the v1 enqueue this rides — its binary-queue clause holds no process; Redis holds the state.

    The honest seam (INV6): v2 interop is the EchoMQ fleet's own — stock BullMQ clients cannot speak v2, by design, not by omission. Today only the Elixir reference speaks v2. The Go port (apps/echomq-go) must implement v2 to join the fleet — not yet. The proposed echomq-node, a first-party Node.js runtime, is proposed; it does not exist. The probe reports the cross-runtime status on every surface that claims interop, never smoothed. Step the round-trip below to see each direction and the honest fleet status.

    The fleet round-trip · first-party runtimes only

    # the shipped enqueue the fleet probe rides — EchoMQ.Queue.add/4 (v1 line)
# @spec add(atom | pid | String.t, job_name, job_data, keyword) ::
#         {:ok, Job.t} | {:error, term}
# the binary-queue clause: no process — Redis holds the state
def add(queue, name, data, opts) when is_binary(queue) do
  conn = Keyword.fetch!(opts, :connection)
  ctx  = Keys.new(queue, prefix: prefix)        # v1: builds emq:queue:… verbatim
  job  = Job.new(queue, name, data, opts)
  add_job(conn, ctx, job)                       # emq.1 ships the v2 form: emq:{payments}:wait
end
    R2 · the pattern

    An invariant must be checked, not promised. A multi-key change is correct because Lua makes it indivisible; "the deployment conforms" is correct only when a probe captures the transition and asserts it.

    emq.1 · the implementation · the 2.0 break

    emq.1 ships the v2 conformance probe (emq:{q}:*-only, one hashtag, zero emq:*, every key declared) and the fleet-interop probe — the interop the fleet's own, the cross-runtime status honest (INV6).

    Interop is the fleet's own. The conformance probe proves a deployment conforms to 2.0; the fleet-interop probe proves a job crosses between first-party runtimes — stock BullMQ clients cannot, by design.

    The v1 flaw the break resolves

    The conformance probe asserts "every key declared" because the v1 line, today, reaches keys it never declares. In moveToActive-11.lua the worker fetch loop builds a job key with local jobKey = keyPrefix .. jobId (:148), then a lock key with local lockKey = jobKey .. ':lock' (:162) — both from an ARGV keyPrefix, neither in the script's KEYS[]. The caller sends only the declared list: EchoMQ.Scripts.execute_raw/4 issues ["EVALSHA", sha, num_keys | keys ++ encoded_args] (scripts.ex:256), where num_keys = length(keys) — the concatenated keys are not in it. Roughly half the v1 set, about 24–26 of the 50 scripts, share the pattern.

    On a single-threaded engine this is invisible — but it sits outside the engine scripting contract (Valkey's own guidance: every key a script accesses arrives as an input key argument), and a key the store cannot see before execution is a key no placement or locking discipline can reason about. The inspector below walks each anchor to its consequence. emq.1 ships the declared-keys v2 set as the root resolution (D2): every operand key in KEYS[], the runtime-pick core redesigned so a popped id resolves inside the queue's native {q} slot.

    The v1 flaw · anchor → consequence → resolution

      The 2.0 fork — EchoMQ leaves the BullMQ wire

      The v1 scripts reach keys they never declare (moveToActive-11.lua:148,162; the caller's declared list is scripts.ex:256) — roughly 24–26 of 50 — outside the engine scripting contract. emq.1 ships the declared-keys v2 set as the root resolution, and the v2 conformance probe asserts it: emq:{q}:*-only, one hashtag per queue, zero emq:*, every key declared. Stock BullMQ clients cannot speak v2, by design — the interop is the fleet's own.

      Recap — conformance, not compatibility

      EchoMQ 2.0 does not check compatibility with BullMQ; it checks conformance to its own protocol. emq.1 ships two probes: the v2 conformance probe (add → process on a v2 queue asserts emq:{q}:*-only · one hashtag per queue · zero emq:* · every key declared · meta.version = echomq:2.0.0) and the fleet-interop probe (the Elixir reference ⇄ the Go port once it ports v2 ⇄ the proposed echomq-node). The WHY is the v1 undeclared-keys inheritance — moveToActive-11.lua:148,162, about 24–26 of the 50 scripts, the caller's declared list at scripts.ex:256 — keys reached outside the engine scripting contract; 2.0 declares every key. The honest seam: stock BullMQ clients cannot speak v2, by design. The next page in the chapter is the E3 landing — the full break emq.1 ships.

      References

      Where the v1 flaw the break resolves is documented, the engine contract the declared-keys set honors, and the chapter's own pages.

      Sources

      Related in this course