jonnify·bcs /bcs/bus/state-machine/the-morgue-and-the-reaper

B3.3 · dive 3 — the morgue and the reaper

Crash recovery is one zset scan on the server's clock.

L5 gates the cap: attempts 2 against max 2 is the morgue — state dead, last_error kept, and the dead set browses in mint order like everything else. L6 gates the crash story end to end: a 40 ms lease expires unanswered; reap returns the orphan to pending and the next claim holds token 2. No heartbeat protocol, no consumer registry — lateness is already written into the active set's scores.

Source: content/bcs3.3.md, quoting bcs_rung_3_3_check.out; reap and promote are the two pumps a supervisor will drive on a cadence.

The cap, exercised. One fixed job under max_attempts 2. Run a retry at each count; the readout names where the row lands and why.
retry(id, token, err) the fork: attempts against max 2 attempts 1 < max 2 schedule state scheduled · the second life attempts 2 against max 2 dead the morgue · last_error kept
readout · Run a retry. The fixed queue class carries max_attempts 2; the fork weighs the counter against the cap before parking anything.

Static verdicts — retry at attempts 1 → the schedule, state scheduled, a second life behind the delay · retry at attempts 2 → the morgue: state dead, last_error kept, the dead set browsing in mint order.

§1

The transcript

bcs_rung_3_3_check.out · verbatim · this dive reads L5 and L6
content/echo_data/runtimes/elixir/bcs_rung_3_3_check.out · verbatimfrozen
L1 surface ok -- the machine's surface: claim, complete, retry, promote, reap join enqueue, browse, pending_size -- five new verbs, every transition one script
L2 happy ok -- claim hands out the oldest job with a server-clock lease and fencing token 1; complete with the right token retires the row -- nothing remains
L3 fence ok -- a stale token is refused on the wire: EMQSTALE; the lease holder's work survives the zombie's complete
L4 schedule ok -- retry parks the job in the schedule, promote moves the due back to pending, and the next claim hands token 2 -- one job, two lives, one counter
L5 dead ok -- attempts 2 against max 2 is the morgue: state dead, last_error kept, and the dead set browses in mint order like everything else
L6 reap ok -- a 40 ms lease expires unanswered; reap returns the orphan to pending and the next claim holds token 2 -- crash recovery is one zset scan on the server's clock
PASS 6/6
§2

The morgue

content/bcs3.3.md · What · Who · When

L5 gates the cap: attempts 2 against max 2 is the morgue: state dead, last_error kept, with the dead set lex-browsable in mint order. The dead set is score-zero like pending, so the morgue browses newest-first like everything else in this series — the order theorem's third appearance on the bus. Set max_attempts per queue class — a fill confirmation and a report render do not deserve the same patience.

The morgue's reader is the postmortem reader, who browses the morgue newest-first and finds last_error waiting on every row. The row that failed twice keeps the words of its last failure — the queue's contribution to the incident review is already written down.

§3

The reaper

content/bcs3.3.md · What · When

L6 gates the crash story end to end: a 40 ms lease expires unanswered; reap returns the orphan to pending and the next claim holds token 2 -- crash recovery is one zset scan on the server's clock. No heartbeat protocol, no consumer registry — the active set is scored by lease deadline, so lateness is already written into its scores, and recovery is one range scan against the server's TIME. The next claim mints token 2 by the same HINCRBY, so a late complete carrying token 1 meets the fence from the previous dive: refused on the wire, EMQSTALE.

The crash story, on the clock. One fixed job, one 40 ms lease, one dead consumer. Step through L6's four beats; the readout tracks the clock and the sets.
readout · Press step to run the crash story — claim, the silence, the reap, the next claim — on the server's clock.

Static walk — 1) claim: token 1, a 40 ms lease; the active set scores the job at its deadline · 2) the consumer dies; the lease expires unanswered · 3) reap: one zset scan by score against the server's clock; the orphan returns to pending · 4) the next claim holds token 2; a late complete with token 1 answers EMQSTALE.

Size the lease above the worst legitimate work time and well below your retry tolerance — the lease is a crash detector, not a deadline for excellence.

§4

Completion deletes, and the boundaries

content/bcs3.3.md · Decisions · Boundaries

Completion deletes. A successful job leaves no residue by default — the row is gone, the receipt is the caller's reply. The audit and event trail is the lane the manuscript assigns to B3.5 · Bus Meets Stores, where results flow back into the stores as first-class writes — pre-stated here so deletion reads as a decision rather than an omission.

NO EXTENSION

No lease extension yet

A long job either gets a long lease or gets split, and the heartbeat verb is a carried follow-up with its own review gate.

THE CAP

Reap caps at one hundred per call

And promote at the caller's batch — both pumps need a driver, which is the loop the manuscript assigns to B3.4 · Fair Lanes, so this chapter ships beats without a metronome.

JUDGMENT

Backoff curves live above the wire

Backoff policy — exponential, jittered, class-dependent — is application judgment, and Lua is the wrong home for judgment.

NO PRECEDENT

The exception does not generalize

A script wanting keys it cannot declare and cannot derive from the queue's own prefix is a design smell, not a precedent.

§5

References

sources · related

Sources

Related