jonnify·bcs /bcs/bus/state-machine/the-fencing-token

B3.3 · dive 2 — the fencing token

One job, two lives, one counter.

L3 stages the zombie: a consumer holding token 1 watches an impostor's complete with token 99 earn EMQSTALE — the row still active, the real holder unharmed. L4 runs the full round trip: retry parks the job in the schedule, promote moves the due back to pending, and the next claim hands token 2. The same integer that counts the attempts fences each life against the last.

Source: content/bcs3.3.md, quoting bcs_rung_3_3_check.out; the verification half is two lines of Lua and one pattern match per transition.

The fence, simulated. One job row at attempts 1, two callers presenting tokens. Run either complete; the readout carries the wire's actual answer.
the fence · presented token == attempts? the lease holder presents token 1 the zombie presents token 99 the job row state active · attempts 1
readout · Run a complete. The fence reads attempts — the counter stands at 1 — and compares it with the presented token before touching anything.

Static verdicts — token 99 against attempts 1 → EMQSTALE complete token mismatch, the row stays active, the lease holder's work survives the zombie's complete · token 1 against attempts 1 → the verify passes and the row retires — nothing remains.

§1

The transcript

bcs_rung_3_3_check.out · verbatim · this dive reads L3 and L4
content/echo_data/runtimes/elixir/bcs_rung_3_3_check.out · verbatimfrozen
L1 surface ok -- the machine's surface: claim, complete, retry, promote, reap join enqueue, browse, pending_size -- five new verbs, every transition one script
L2 happy ok -- claim hands out the oldest job with a server-clock lease and fencing token 1; complete with the right token retires the row -- nothing remains
L3 fence ok -- a stale token is refused on the wire: EMQSTALE; the lease holder's work survives the zombie's complete
L4 schedule ok -- retry parks the job in the schedule, promote moves the due back to pending, and the next claim hands token 2 -- one job, two lives, one counter
L5 dead ok -- attempts 2 against max 2 is the morgue: state dead, last_error kept, and the dead set browses in mint order like everything else
L6 reap ok -- a 40 ms lease expires unanswered; reap returns the orphan to pending and the next claim holds token 2 -- crash recovery is one zset scan on the server's clock
PASS 6/6
§2

The fence, performing

content/bcs3.3.md · Why · What · How

A queue that hands out work without leases leaks jobs the first time a consumer dies; a queue with leases but no fencing corrupts data the first time a consumer pauses — the classic failure where a process stalls past its lease, the job is handed to another worker, and then the first wakes up and completes stale work over fresh state. Kleppmann's analysis named the cure — a lock service that "generates strictly monotonically increasing tokens", with the resource refusing any token that goes backwards. The attempts counter is that token: monotonic per job because only the claim script increments it, and a second token field would be a second authority.

Complete and retry both verify the token before touching anything, and refuse backwards tokens with the part's second wire class. The client half of fencing is two lines of Lua and one pattern match per transition:

local att = redis.call('HGET', KEYS[2], 'attempts')
if att ~= ARGV[2] then return redis.error_reply('EMQSTALE complete token mismatch') end
{:error, {:server, "EMQSTALE" <> _}} -> {:error, :stale}

EMQSTALE joins EMQKIND: the wire-class family grows by one, and every refusal a bundle script issues leads with its class. And read {:error, :gone} from complete as the calm answer it is: the row was already retired, usually by a racing twin of yourself, and at-least-once semantics promised exactly this possibility.

§3

Two lives, one counter

content/bcs3.3.md · What · When

L4 runs the full round trip: retry parks the job in the schedule (state scheduled, pending empty), promote moves the due back after the delay, and the next claim hands out token 2 — the committed line reads one job, two lives, one counter, which is the chapter in six words: the same integer that counts the attempts fences each life against the last.

The round trip, walked. One fixed job through its two lives. Step through; the readout tracks the state and the counter at each transition.
readout · Press step to walk one job through claim → retry → promote → claim and watch the one counter carry both duties — counting the lives and fencing them.

Static walk — 1) claim: token 1 minted, state active — the first life · 2) retry with token 1: the verify passes, the job parks in the schedule (state scheduled, pending empty) · 3) promote: the due job moves back to pending · 4) claim: HINCRBY mints token 2, state active — the second life, fenced against the first.

Choose the retry delay at the caller: the script takes a literal milliseconds argument by design, because backoff policy (exponential, jittered, class-dependent) is application judgment and Lua is the wrong home for judgment.

§4

The Go side

content/bcs3.3.md · How

Nothing ports because nothing needs to: the five scripts are the contract, same bytes, same SHA1, same semantics from any client. A Go consumer is a loop around the same EVALSHA calls:

id, payload, token, ok := claim(conn, q, lease)
if !ok { time.Sleep(idle); continue }
if err := work(payload); err != nil {
    retry(conn, q, id, token, backoff(attempt), maxAttempts, err.Error())
} else {
    complete(conn, q, id, token)
}
§5

References

sources · related

Sources

Related