jonnifyechomq

E2.04.2 · The three runtimes · dive 2

The Go gaps

The port speaks the same wire — but it is partial, and a depth course names the gaps rather than smoothing them.

Three things, told straight. The non-atomic add path: where the Elixir reference runs one atomic addStandardJob-9.lua, the Go port runs two separate commands. The attemptsMade-vs-atm nuance: Go writes and reads atm on the hash — only its external JSON tag is attemptsMade, an L4 surface, not a wire bug. And the cluster validator the Go port does carry: GetClusterSlot + ValidateHashTags, the CRC16-modulo-16384 rule. Step the add comparison on the right.

Add a standard job · one atomic script vs two commands

step 1 — step 2 —

Gap one — the non-atomic add path

On the Elixir reference, enqueuing a standard job runs one atomic Lua script — addStandardJob-9.lua, via Scripts.add_standard_job (scripts.ex line ~435). The job hash is written and the id pushed onto the wait list in a single round-trip that either fully happens or does not:

# scripts.ex — one atomic script for the whole add
execute(conn, :add_standard_job, keys, args)   # addStandardJob-9.lua, 9 declared keys

The Go port's add path runs two separate commands: addJobToQueue writes the hash with HSet (queue_impl.go line ~105), and then enqueueJob pushes the id with a separate LPush (or ZAdd for delayed/priority, queue_impl.go line ~137):

// queue_impl.go — addJobToQueue: first command
return q.redisClient.HSet(ctx, key, fields).Err()

// queue_impl.go — enqueueJob: second, separate command
return q.redisClient.LPush(ctx, q.keyBuilder.Wait(), job.ID).Err()

Where the protocol expects one atomic script, the Go port issues two commands. A failure between them can leave a job hash with no list entry. This is the non-atomic critical path — a real porting gap, reported, not glossed. (The Go port's fetch path is atomic: the worker pickup runs the moveToActive Lua script — worker_impl.go line ~112 — so the gap is on the add side, not the pickup side.)

One atomic script versus two separate commands. The Elixir add either fully happens or does not; the Go add has a window between HSet and LPush. That is the non-atomic critical path.

The nuance — attemptsMade vs atm

A common worry is that the Go port writes the wrong field name. It does not. The Go port writes and reads atm on the Redis hash, exactly as the wire requires:

// queue_impl.go ~102 — WRITE the hash field as "atm"
"atm": job.AttemptsMade,

// script_runner.go ~160 — READ the hash field "atm" back
if attemptsMade, ok := data["atm"]; ok {
  fmt.Sscanf(attemptsMade, "%d", &job.AttemptsMade)
}

Only the Go struct's external JSON tag is attemptsMade (job.go line 22):

// job.go:22 — the JSON tag is an L4 serialisation surface, NOT the wire
AttemptsMade int `json:"attemptsMade"`

That tag is an L4 surface — how a Go program serialises the struct to its own callers — not the wire. The Redis field round-trips as atm in both directions, so a job written by Go and read by Elixir (whose EchoMQ.Job.from_redis/4 reads the same compressed fields) agrees on the field. This was convergently verified twice: it is a serialisation tag, not a wire bug.

The Go port reads and writes atm on the wire; only its JSON output is named attemptsMade. That is an L4 tag, not a wire difference — the interop holds.

What the Go port does carry — the cluster validator

The Go port carries a real Redis Cluster slot validator the Elixir runtime lacks. pkg/echomq/cluster.go implements GetClusterSlot and ValidateHashTags, with RedisClusterSlots = 16384:

// cluster.go — the slot is CRC16 of the {hashtag}, modulo 16384
const RedisClusterSlots = 16384

func GetClusterSlot(key string) int {
  hashKey := key                          // extract the {hashtag} content if present
  crc := CalculateCRC16([]byte(hashKey))
  return int(crc) % RedisClusterSlots
}

func ValidateHashTags(keys []string) (bool, int, []int) {
  // true iff every key hashes to the SAME slot
}

The rule is slot = CRC16(hashtag) % 16384; keys that share a hashtag share a slot, which is the precondition for running a multi-key Lua script on a Redis Cluster. The checker below applies the rule to a fixed pair of keys: type a hashtag and read whether two keys built with it land in the same slot. It demonstrates the rule, not a specific slot number.

ValidateHashTags · same hashtag ⇒ same slot

emq:{orders}:wait emq:{orders}:active

The cluster validator is a genuine Go-side addition: it confirms a multi-key script's keys share one slot before running it. The rule is CRC16(hashtag) % 16384 — same hashtag, same slot.

The Golden Rule — gaps measured against the shared wire

The wire is the same for every runtime — the emq: keyspace and the field names (including atm) are L1, and a multi-key transition is one atomic Lua script, L2. The Go port's gaps live above that line, in L3/L4: the add path runs two commands instead of the atomic script, and the attemptsMade JSON tag is an L4 serialisation choice over the atm field. The cluster validator it adds is also L3 — the slot rule it checks is the protocol's.

the protocol (immutable L1/L2)

The emq: keyspace and the field names — including atm on the hash (L1); a multi-key transition is one atomic script (L2) — frozen and shared.

its Go runtime (variable L3/L4)

A partial port — the add path runs two separate commands (the non-atomic gap), the attemptsMade JSON tag is an L4 serialisation choice over the atm field, and it adds a CRC16-based cluster slot validator the reference lacks.

The Go port is honestly partial: a non-atomic add path is a real gap, the attemptsMade/atm mismatch is an L4 tag and not a wire bug, and the cluster validator is a genuine Go-side addition. A depth course names all three.

Next: with both runtimes covered, how does a job written by one get processed by the other? E2.04.3 · cross-runtime interop walks the round-trip and the v2 trajectory.

The 2.0 fork — EchoMQ leaves the BullMQ wire

The gaps above are measured against the v1 line (frozen at 1.3.0) — the BullMQ wire. emq.1 ships EchoMQ 2.0, and the Go port must implement 2.0 to stay in the fleet: the v2 declared-keys script set, the emq:{q}:… keyspace, and the transparent {q} hashtag. The Go port's gap list is its porting work-list for v2 — the non-atomic add path becomes a v2 atomic script, and the cluster validator it already carries is exactly the placement check the v2 hashtag design formalises. emq.1's fleet-interop probe tracks the port's v2 status honestly until it lands.

References

The atomic add and fetch scripts, the cached-script dispatch, and the Redis Cluster slot scheme the validator implements.

Sources

Related in this course