R2.04.2 · Cross-shard consistency — dive 2
Detect, then order.
A shared token catches the tear. A version field adds the second answer the read needs — which write to keep. On a mismatch, accept the higher version, rebuild the lower from it.
Grounding: echo/apps/echo_mq · EchoMQ.Jobs claim → @claim (jobs.ex:126, HINCRBY attempts), complete → @complete (jobs.ex:139, EMQSTALE token mismatch) — attempts is the applied version token. The protocol fence is {emq}:version (echo_wire/connector.ex:33).
Static verdicts — profile newer → v12 > v11, keep profile, regenerate settings · settings newer → v12 > v11, keep settings, regenerate profile. Last-write-wins, resolved at read time.
From detect to order
cross-shard-consistency.md.txt · Version-Stamped ValuesThe shared-token detector from the previous dive returns one bit: the tokens are equal, or they are not. That catches a torn write, but it is not enough to repair one without help. On a mismatch the tokens alone do not name which write landed last. Stamp the values with something ordered and the read gains a second answer. Prefix each value with a version field ahead of the token:
SET user:123:profile "{timestamp}:{token}:{payload}"
SET user:123:settings "{timestamp}:{token}:{payload}"
On read, parse the version off each value. Equal tokens still mean the pair is whole. On a mismatch, the higher version names the write to keep — accept it, and regenerate the lower-versioned side from it. The recovery policy turns from consult the source of truth into accept newest — last-write-wins, resolved at read time with no extra round trip:
# the recovery rule on a mismatch — one comparison if profile_version > settings_version: regenerate_settings_from_profile() # profile newer — keep it, rebuild settings else: regenerate_profile_from_settings() # settings newer — keep it, rebuild profile
Accept-newest assumes the higher-versioned side is a complete, correct write the lower side can be rebuilt from. Where both sides hold partial state, the safer recovery is to re-fetch the authoritative source and rewrite both, which the version does not replace.
A token detects the tear; a version also names the write to keep. On a mismatch the higher version wins, and the read resolves last-write-wins instead of returning an undecided pair.
Timestamps skew; INCR does not
cross-shard-consistency.md.txt · Monotonic VersionsA wall-clock timestamp is the cheapest version field: every host already has a clock. Its guarantee is only as good as those clocks. Two hosts that disagree by a few hundred milliseconds can stamp writes in the wrong relative order — a write that truly happened second can carry the lower timestamp because the host that issued it ran behind. Under skew the accept-newest rule then keeps the value written first. Clock skew across a fleet is the normal condition; NTP narrows it but never to zero.
For ordering that does not depend on any clock, mint the version from one monotonic source.
INCR user:123:version returns a counter that rises by exactly one on every call and is served by
a single key, so two writes can never receive the same version and a later write always receives a higher
one:
version = INCR user:123:version
SET user:123:profile "{version}:{payload}"
SET user:123:settings "{version}:{payload}"
The trade is explicit — a timestamp costs nothing and orders correctly only while clocks agree; an
INCR version costs one round trip and orders correctly always. The figure injects clock skew and
runs both resolvers over the same two writes. The true order is fixed: settings is written first, profile
second, so profile is the newer value.
Static verdicts — clocks agree → both resolvers pick profile (the newer value) · settings +250 ms → the timestamp resolver picks settings (wrong), the INCR resolver still picks profile (correct), because the counter orders by one source.
A wall-clock timestamp orders correctly only while the clocks agree; a monotonic
INCR version orders by one source and resolves the same torn pair correctly even when a clock runs
ahead.
Applied — attempts is the version token
echo_mq/jobs.ex:126,139 · connector.ex:33
EchoMQ does not stamp a free-standing version on its co-located queue keys, because the tear a version
would resolve cannot occur there: a queue's keys share one slot and the move runs as one Lua EVAL
(the previous dive). But the same idea — a monotone field that names the current writer — is
exactly how EchoMQ fences a stale worker, and it is real code. When EchoMQ.Jobs.claim/3 leases a
job, the @claim script advances the job's attempts field by one with
HINCRBY and returns the new value to the worker. attempts is the fencing
token: it only ever rises, by one, per claim.
A worker carries the att it was handed. When it finishes, EchoMQ.Jobs.complete/4
runs the @complete script, which compares the stored attempts against the token the
worker presents:
local att = redis.call('HGET', KEYS[2], 'attempts')
if not att then return 0 end
if att ~= ARGV[2] then
return redis.error_reply('EMQSTALE complete token mismatch')
end
← the stale writer is refused; only the current attempts holder may complete
If a worker's lease expired and the job was re-claimed by another, the row's attempts has
advanced; the first worker's token is now stale, and its complete is refused with
EMQSTALE complete token mismatch (the same fence guards @retry). That is
detect-and-order applied to a queue: the higher attempts is the current holder, the lower one is
the stale writer, and the engine resolves the conflict by rejecting the stale side rather than silently
overwriting.
Notes on ValkeyINCR and HINCRBY are atomic
single-key increments — two writers can never receive the same value, which is what makes
attempts a sound monotone fence —
valkey.io/commands/incr.
Two versions, two keys — the EchoMQ course, in depth
The protocol's own version fence is a different number on a different key: {emq}:version
→ the wire string below, claimed or verified by EchoMQ.Connector before the first command.
It guards the protocol, not a single job. The in-depth treatment of both the attempts fence and
the version fence is the dedicated EchoMQ course.
@wire_version "echomq:2.0.0"
← GET {emq}:version, claimed or verified before the first command (connector.ex:465)
A version field on each related value detects the tear and orders it: on a mismatch the higher version is the write to keep, resolved at read time as last-write-wins.
The applied version token is attempts — a monotone HINCRBY in
@claim. A worker holding a stale attempts is refused at complete /
retry with EMQSTALE … token mismatch.
A token detects the tear; a monotonic version also tells you which write to keep. EchoMQ's
attempts is that version applied to a queue — the higher value is the current worker, the
lower one is fenced off.
Recap — order the tear
next → R2.04.3A bare token catches a tear but cannot order it. A version field adds the second answer: on a mismatch the
higher version wins. A wall-clock timestamp orders cheaply but skews; a monotonic INCR orders by
one source and resolves correctly always. EchoMQ applies exactly this as the attempts fence:
only the current token holder may complete a job. What remains is making an incomplete write invisible
— the next dive's commit marker.
References
sources · relatedSources
- Redis — INCR — the atomic monotonic counter;
INCR user:versionmints a version that orders writes correctly regardless of any host's clock. - Valkey — INCR — the engine's own atomic increment; the same monotone source EchoMQ's
HINCRBY attemptsfence is built on. - Redis — SET — the per-value stamped write
SET key "{version}:{payload}". - Redis — Cluster specification — hash tags and the 16384 slots the colocation relies on.
Related in this course
- R2.04 · Cross-shard consistencythe module hub
- R2.04.1 · Torn writesthe bare token this upgrades
- R2.04.3 · Commit markersthe next dive
- R2 · Coordinationthe chapter
- /echomqthe attempts & version fence in depth
- /elixir · CQRSthe single-writer engine