jonnify·redis /redis-patterns/coordination/atomic-updates

R2.01 · Coordination — module hub

Atomic updates.

Ensure data integrity with atomic read-modify-write operations: WATCH/MULTI/EXEC for optimistic locking, Lua scripts for complex logic, and shadow-key patterns for safe bulk updates. EchoMQ moves every job between states with one inline Lua EVAL — the move that cannot be torn.

Grounding: echo/apps/echo_mq/lib/echo_mq/jobs.ex · enqueue/4 → the @enqueue script · echo_wire/connector.ex eval/5 (EVALSHA-first) — the v2 law: every key a script touches is declared in KEYS. Engine: Valkey.

The lost update — two withdrawals on one balance. Select a mode to light its path and read the final balance and how many updates were lost.
Client Awithdraw 30 Client Bwithdraw 30 balancestart 100 GET / SET (both read 100) one EVAL — serialized, 100 → 70 → 40
readout · Select a mode. Two GET/SET clients lose one withdrawal (ends 70); one atomic move serializes them (ends 40, correct).

Static verdicts — two GET/SET clients → both read 100, the last SET wins, ends 70, one update lost · one atomic move → the two serialize, 100 → 70 → 40, nothing lost.

§1

The lost update

atomic-updates.md.txt

Two clients withdraw from one account. Each runs GET balance, subtracts its amount, and runs SET balance. With a starting balance of 100 and two withdrawals of 30, the correct end is 40. But if the two interleave — both read 100 before either writes — the second SET overwrites the first, and the account ends at 70. One withdrawal is gone, and no error was raised.

The fix is not a faster client; it is an indivisible move. Make the read and the write happen as one unit and nothing can interleave between them. The rest of this module is the three Redis tools that do that, and the single Lua EVAL EchoMQ built every state transition from.

A lost update is silent. The cure is to make the read and the write one move; the three tools that follow are how Redis does it.

§2

Optimistic locking with WATCH

atomic-updates.md.txt · Pattern 1

WATCH gives check-and-set semantics. A client watches a key, reads it, opens a MULTI/EXEC block, and queues its writes. If any watched key changes between the WATCH and the EXEC, the transaction aborts — EXEC returns nil — and the client retries the whole read-modify-write. It is optimistic: it holds no lock and assumes no conflict, paying only when a conflict actually lands.

WATCH account:123:balance
balance = GET account:123:balance
MULTI
SET account:123:balance <balance - 100>
EXEC                                  # nil if balance changed since WATCH -> retry

A retry loop bounds the attempts (max_retries) and raises a conflict error if it runs out. Multi-key WATCH watches several keys for one update across them — WATCH inventory:sku123 order:456:status, then DECRBY the stock and SET the order status in the same EXEC. On a cluster all watched keys must sit on one slot; a hash tag co-locates them. The first dive walks the loop and the multi-key case.

Be honest about the applied form: EchoMQ does not run WATCH/MULTI/EXEC for its transitions. It replaced the optimistic-locking loop with a single Lua script — one round trip, no retry. WATCH is taught here as the contrast.

WATCH trades a retry for never holding a lock: it lets the conflicting writers race, and only the loser pays, by trying again.

§3

Lua scripts for atomic logic

atomic-updates.md.txt · jobs.ex:14

A Lua script runs atomically — no other command runs between the script’s start and its end. That removes the retry loop entirely: the conditional check, the reads, and the writes happen as one indivisible unit. A compare-and-set is four lines; the queue’s admission is a kind check, a duplicate refusal, a row write, and a pending insert — all one EVAL.

-- compare-and-set: set only if the current value matches the expected one
if redis.call('GET', KEYS[1]) == ARGV[1] then
  redis.call('SET', KEYS[1], ARGV[2])
  return 1
end
return 0

This is the tool a busy queue reaches for, because its state moves are conditional and span several keys. EchoMQ’s admission law is one inline Lua script: EchoMQ.Jobs.enqueue/4 runs @enqueue through EchoMQ.Connector.eval/5 — a kind check, an EXISTS duplicate refusal, the HSET row write, and the ZADD pending insert land on the server in one atomic step or not at all.

-- @enqueue (echo_mq/lib/echo_mq/jobs.ex:14) — admit by kind, refuse duplicates, write the row + pending entry.
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])
redis.call('ZADD', KEYS[2], 0, ARGV[1])
return 1

The script declares both keys it touches — KEYS[1] the job row, KEYS[2] the pending set — in KEYS; ARGV carries values only. That is the v2 law (EchoMQ.Script.new/2): every key a script touches is declared. Because both keys share the {orders} hashtag (EchoMQ.Keyspace.queue_key/2 braces the queue), they land on one slot, so the multi-key write is single-slot legal by grammar — no CROSSSLOT. The second dive grounds this hardest in the real @claim/@complete/@retry scripts.

Notes on ValkeyA Lua script runs atomically: its effects land together or not at all, so the read-branch-write of @enqueue cannot interleave with another client — valkey.io/topics/programmability.

§4

Shadow key + RENAME, and bulk admission

atomic-updates.md.txt · jobs.ex:100

To replace a large, multi-field value without a concurrent read landing on a half-written state, build the new value in a temporary key, then swap it in with one atomic RENAME. RENAME is atomic: a concurrent read returns either the old complete value or the new complete value, never a partial one.

DEL tmp:user:123:cache
HSET tmp:user:123:cache field1 value1
HSET tmp:user:123:cache field2 value2
HSET tmp:user:123:cache field3 value3
RENAME tmp:user:123:cache user:123:cache   # the swap is all-or-nothing

RENAME preserves the source key’s TTL, and both keys of a RENAME or COPY must share a slot. The third dive pairs this with bulk admission — EchoMQ.Jobs.enqueue_many/3 loads the @enqueue script once with SCRIPT LOAD and then pipelines EVALSHA per item, with per-item verdicts (:enqueued, :duplicate, {:error, :kind}) returned in input order — the bulk in one wire flush.

Swap the whole value at once with RENAME, or admit a whole batch with SCRIPT LOAD + pipelined EVALSHA — never leave a reader a half-built state.

§5

The narrower atomic primitives

atomic-updates.md.txt · Patterns 4–7

The same discipline — never read-then-write across a gap — covers four narrower moves:

  • Idempotency keys. SET idempotency:{request_id} "processing" NX PX … runs an operation exactly once. EchoMQ folds the same idea into @enqueue: the EXISTS check returns 0 on a duplicate, so a producer can fire the same enqueue twice on any doubt and the second call answers :duplicate without touching the row.
  • Increment with bounds. A short Lua script caps a counter at a maximum or refuses a decrement below zero. EchoMQ’s monotone counter is the attempts field — incremented with HINCRBY inside the claim script, never decremented, so it is the fencing token every later transition verifies.
  • SET … GET / GETDEL / GETEX. Get-and-modify in one round trip: read-and-replace, read-and-delete, and read-and-re-expire, each a single atomic command.
  • List rotation. LMOVE source destination RIGHT LEFT (and the blocking BLMOVE) pops from one list and pushes to another atomically — the backbone of a reliable queue, which R3 builds on.
§6

When to use each

atomic-updates.md.txt · When

The choice turns on contention and shape. A single-key check-and-set is a Lua one-liner or SET … GET; a same-slot multi-key update is WATCH with a retry; conditional or multi-step logic is a script, where the script is the lock.

ScenarioPattern
Simple CAS on one keySET … GET or a Lua script
Multi-key transaction (same slot)WATCH + MULTI/EXEC
Complex conditional logica Lua script
Large value replacementshadow key + RENAME
Request deduplicationidempotency keys
Counter with boundsa Lua script
Queue operationsLMOVE / BLMOVE
the application · EchoMQ’s atomic admission, against live Valkey

The transition is one EVAL, every key declared

The atomic move is not an aspiration — @enqueue runs through EchoMQ.Connector.eval/5 EVALSHA-first (valkey.io/commands/evalsha), and every key shares the {orders} hashtag so the multi-key write lands one queue in one slot. The committed connector record:

content/bcsA.md · the connector, against Valkey on :6390frozen
script path                   script_loads=1   ← one NOSCRIPT load, then EVALSHA from cache
pipelined EVALSHA             161192 ops/s   ← the declared-keys law at speed
slot law                      emq:{orders}:* → slot 105 == 105
prefix                        17 bytes before the 14-byte payload
§7

The three dives

WATCH → Lua → bulk

Each dive takes one of the three tools, in the source’s order: optimistic locking with a retry loop, the whole transition as one script, and the atomic swap with bulk admission. Read them in order — the retry loop comes first, the script that removes it second, the bulk swap third.

R2.01.1built

WATCH / MULTI / EXEC

Optimistic locking: watch a key, read it, queue the write, and on a conflicting change retry the loop. The account-balance and multi-key inventory cases — the contrast EchoMQ replaced with one EVAL.
R2.01.2built

Lua for logic

The whole transition as one Lua script — no interleaving, no retry. The real @claim/@complete/@retry branching scripts and the EVALSHA dispatch.
R2.01.3built

Shadow key + bulk

Build the new value in tmp: then atomic RENAME; admit many jobs in one wire flush with enqueue_many/3 — SCRIPT LOAD once, then pipelined EVALSHA.
the pattern

A read-modify-write across several keys must be one indivisible move. WATCH/MULTI/EXEC, a Lua script, and an atomic swap are the three Redis tools that make it so.

its EchoMQ application

Every state transition is one inline Lua EVAL — EchoMQ.Jobs.enqueue/4 runs @enqueue through EchoMQ.Connector.eval/5, EVALSHA-first, every key declared in KEYS. No watch, no retry loop.

Redis offers three atomic read-modify-write tools; EchoMQ chose the single Lua EVAL per transition, and proves the pattern is real.

the door · the EchoMQ course

The full transition bundle is the next course

This module cites one excerpt — the @enqueue script — as proof the pattern ships. The full transition bundle (@claim/@complete/@retry/@promote/@reap), the EVALSHA/NOSCRIPT dispatch internals, and the echomq:2.0.0 version fence are the subject of the dedicated EchoMQ course. The Coordination chapter closes with a workshop that makes an Exchange Platform order placement atomic across runtimes; Exchange.Gateway.parse_place/1 mints a branded ORD id at acceptance, and the order is admitted onto the {orders} queue as that one atomic script.

After R2.01, the Coordination chapter continues with distributed locking, the Redlock contrast, cross-shard consistency, and hash-tag colocation. Return to the Coordination chapter to continue, or open the Overview for where Valkey sits under the Exchange Platform. The functional-Elixir & OTP craft behind the echo umbrella is the /elixir course.

§8

References

sources · related

Sources

Related in this course