Batches · dive 03

Batch lease extension

Re-score the many under one clock read. The lease is the active score, so checkpointing many is one script — and it returns the ones it could not extend.

A worker holding several jobs at once checkpoints all their leases together, on one clock. extend_locks/4 re-scores every matching active member under one server-clock read and answers the failed list — the ids whose token was stale or whose row was gone. The lease in this Queue is the active-set score, so extending it is re-scoring the active member; there is no separate lock. Step the batch on the right, then read the two beats and the contrast with the single verb below.

The two beats Single vs batch Pattern & implementation References

Extend four leases · step the members

Batch lease extension, in two beats

Read it the way the protocol is built: the handle (the Elixir verb that gates the ids and runs the script) and the body (the Lua that re-scores the members). The handle is above the line; the body is the protocol.

Beat one — the named handle. extend_locks/4 gates every id at Keyspace.job_key/2 first, flattens the {id, token} pairs into ARGV after the queue base root and the lease window, and runs @extend_locks declaring only the active set. It answers {:ok, failed} — the ids it could not extend.

# echo_mq — EchoMQ.Jobs
# extend_locks/4 re-scores MANY active members under ONE server-clock read.
# Gate every id first (an ill-formed id raises before the wire). Flatten the held
# {id, token} pairs into ARGV after the queue base root + the lease window. The
# declared key is [active]; each per-job key is derived IN-SCRIPT from the base
# root. Answers {:ok, failed} — the ids that could NOT be extended.
def extend_locks(conn, queue, held, lease_ms)
    when is_list(held) and is_integer(lease_ms) and lease_ms > 0 do
  Enum.each(held, fn {id, _token} -> Keyspace.job_key(queue, id) end)   # gate every id

  pairs = Enum.flat_map(held, fn {id, token} -> [id, Integer.to_string(token)] end)
  argv = [Keyspace.queue_key(queue, ""), Integer.to_string(lease_ms) | pairs]

  case Connector.eval(conn, @extend_locks, [Keyspace.queue_key(queue, "active")], argv) do
    {:ok, failed} when is_list(failed) -> {:ok, failed}                 # the ids not re-leased
    other -> other
  end
end

Beat two — the script body. @extend_locks reads the clock once, then walks the ARGV pairs (id, then token), re-scoring each matching member and collecting the rest. KEYS[1] is the active set; ARGV[1] is the queue base root, ARGV[2] the lease window, and ARGV[3..] the (id, token) pairs.

-- the @extend_locks script — one clock read, many members re-scored
-- KEYS[1] is the active set. ARGV[1] is the queue base root, ARGV[2] the lease
-- window in ms; ARGV[3..] is (id, token, id, token, …). The per-job key is
-- DERIVED in-script from the base root — never read out of a data value.
local base = ARGV[1]
local lease = tonumber(ARGV[2])
-- ONE server-clock read shared by every id in the batch.
local t = redis.call('TIME')
local now = t[1] * 1000 + math.floor(t[2] / 1000)
local failed = {}
local i = 3
while i < #ARGV do
  local id = ARGV[i]
  local token = ARGV[i + 1]
  local jk = base .. 'job:' .. id
  -- the fencing check: re-lease only if the row's attempts still matches the
  -- token the caller holds. A moved token (reaped and reclaimed) is NOT extended.
  local att = redis.call('HGET', jk, 'attempts')
  if att and att == token then
    redis.call('ZADD', KEYS[1], now + lease, id)   -- re-score the active member
  else
    table.insert(failed, id)                       -- stale token or gone row — collect it
  end
  i = i + 2
end
return failed

The per-job key is built in-script as base .. 'job:' .. id from the declared queue base root — never read out of a hash value. The single key the script declares is the active set; the active-set score is the lease, so re-scoring it is the whole extension. Every job whose token still matches is re-leased on the same now; every job whose token has moved on — it was reaped and reclaimed, so its attempts no longer matches — is reported in failed, not extended.

Batch lease extension is one script and one clock read: re-lease every matching active member at the shared now + lease, and return the ids whose token had moved on.

Single vs batch — collect, don't error

The single extend_lock/5 re-scores one member. It is token-fenced too, but on a stale token it errors ({:error, :stale}, the EMQSTALE wire class) and on a gone row it answers {:error, :gone}. The batch does not error on a single bad id — it collects it into failed, so one stale lease among many does not abort the others. Pick a job's fate and a verb to read the two answers; the model below is the real return shape of each.

extend_lock/5 vs extend_locks/4
	single `extend_lock/5`	batch `extend_locks/4`
ids	one	many
keys declared	`[active, job_row]`	`[active]` — per-job key derived in-script
clock reads	one	one, shared across all ids
a non-matching token	errors — `{:error, :stale}`	collected — the id joins `failed`
a gone row	errors — `{:error, :gone}`	collected — the id joins `failed`
return	`:ok` \| `{:error, …}`	`{:ok, failed}`

A job's fate, two verbs · read both answers

The single verb errors because there is nothing to collect — one id, one answer. The batch collects because it has many ids and one result: a stale lease becomes an entry in failed, and the rest are re-leased on the same clock.

Pattern & implementation

the pattern (Redis Patterns Applied)

A batch over a sorted set should read its reference point — the clock — once, so every member is scored against the same instant, not a drifting one. Reliable queues teaches the reliable-claim move; the one-clock-many-members angle is the near side of the door.

the implementation (echo_mq)

EchoMQ.Jobs.extend_locks/4 runs @extend_locks, which reads the server TIME once and re-scores every active member whose token matches, returning the failed list — where the single extend_lock/5 re-scores one member and errors on a stale token.

The pattern says: read the clock once, score the many against it. The implementation is extend_locks/4 — one TIME, many re-scores, and the failed list for the ones that moved on.

Recap — the many, under one clock

Batch lease extension re-scores the many under one clock read: gate every id, walk the held pairs, re-lease each matching active member at the shared now + lease, and return the ids that could not be extended. The single verb errors on a bad id; the batch collects it — one stale lease does not abort the rest. That closes the Batches module: admit, compose, and extend many in one flush.

Back to the module: Batches — the three bulk verbs and the one rule they share. The next module in The Queue covers the control plane beyond enqueue and claim.

References

Where the batch lease extension is grounded, and the commands the script uses.

Sources

Valkey — ZADD — the active-set re-score that extends a lease (the score is the deadline).
Redis — EVALSHA — the run-by-SHA dispatch the batch script runs under.
Valkey — HSET — the row write whose attempts field is the fencing token the script checks.
Valkey — Documentation — the substrate of record EchoMQ is backed by.

Related in this course

Batches — the module this dive belongs to.
Bulk flows — compose many flows in one call, fail-closed per flow.
Enqueue many — admit many jobs in one flush.
Scripts are the protocol — a transition read in two beats.
redis-patterns · Reliable queues — the near side of the door.