jonnify·bcs /bcs/trading/log-and-ledger/the-journal-and-the-shadow

B8.2 · dive 1 — the journal and the shadow

Durability beside the bus, not inside it.

A trading platform's log answers one obligation: every fill must survive the bus. The bus is volatile by decision D-2 — a restart erases its queued jobs — so durability lives beside the bus. The as-built EchoCache.Journal is that durability: a per-group SQLite file holding two memories, the outbox of intents at the writer's edge and the last applied version per name at the applier's edge. The committed gate fixes the schema: one journal file per group, named by the group's branded id ... the schema carries the two memories.

Source: the journal is real Elixir at echo/apps/echo_cache/lib/echo_cache/journal.ex, taught in content/bcs4.4.md, its record bcs_rung_4_4_check.out closing PASS 6/6. The book that owns one per instrument, Exchange.Book, is PROPOSED (docs/trading/) — taught as design, never as a measured number.

The two-memories schema viewer. Select a table to read its columns and its role over a fixed two-name example — the schema's own shape, computed from the record.
intents · the outbox applied · the last word one file per group, named by the group's branded id · a non-branded group is refused at the door
readout · Select a table to read its columns and role. The intents table is the transactional outbox — record before the enqueue, mark after; the applied table is the lane's last word per name, and it outlives the table, the node, and the bus.
§1

The two memories, in one file

echo/apps/echo_cache/lib/echo_cache/journal.ex · verbatim surface

The intents table is Richardson's transactional outbox: seq, the recorded job_id, the name_id, the version, an enqueued flag, a recorded_at — the cargo law in a schema, nothing else. The applied table is the lane's last word: name_id primary key, the newest version, its seq. The applied table lives in the file, so it survives the table, the node, and the bus.

The as-built surface, quoted from source — two writer edges and an outbox verb:

# the writer's first edge: record the intent before the bus hears it
record(j, job_id, name_id, version)
# the writer's second edge: the bus accepted this intent
mark_enqueued(j, job_id)
# group commit: one WAL append amortized across a batch
record_many(j, triples)
# the outbox in one verb: mint a job id, record, enqueue, mark
intend_and_enqueue(j, conn, name_id, version)
# a consumer handler wiring the job lane through the journal's memory
handler(j, table)
replay(j, conn)   # recovery beside the bus
compact(j)        # retire every intent whose name carries an applied version at least as new

Two doors, one law: a non-branded group is refused (a non-branded group is refused at the door). One journal file per group, one owner process per journal — the single writer of Chapter 4.3 as a storage discipline, which is what makes the five hot statements prepared once at journal start and rebound per call trivially safe.

§2

Both crash seams, closed by what already exists

bcs_rung_4_4_check.out · H2 · verbatim

The writer's flow is record, enqueue, mark — two crash seams. A death before the enqueue leaves a pending intent that replay enqueues. A death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it. Neither seam needs new machinery: the bus already deduplicates a job id at admission, and newer-wins already makes a re-applied version a no-op.

The two-seams walk. Step a recorded intent through the writer's flow, choose a death point, and read what replay does — re-enqueue, or :duplicate at admission. The verdict is the H2 record's own.
recordintent written enqueueonto the bus markenqueued=1 seam 1 · death here → replay enqueues seam 2 · death here → :duplicate at admission
readout · A recorded intent sits at record. Advance it through enqueue and mark, or choose a death point and run replay. Coverage by the applied memory makes a fully-marked intent's replay a no-op.
content/echo_data/runtimes/elixir/bcs_rung_4_4_check.out · the seams this dive walksfrozen
derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op
H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0}
PASS 6/6
§3

The journal's WAL is beside the bus, not inside it

content/bcs4.4.md · D-2

The law's wording is exact: the bus gets no WAL; the journal, which is not the bus, gets SQLite's. D-2 stands untouched — the bus stays volatile, restart semantics stay replay, and durability lives in a file the bus has never heard of. The journal opens PRAGMA journal_mode=WAL, synchronous=NORMAL, busy_timeout=5000 — the last because the shadow takes brief write locks at checkpoint and the journal waits rather than errors.

mark_enqueued is a separate call on purpose: folding it into the enqueue would erase the seam the drill exists to cover. The outbox pattern's candor is that the seams are real, named, and closed by replay rather than wished away. The dual-write trap waits on either side of the obvious fix — write the store then enqueue and a crash loses the message; enqueue then write and a crash announces a write that never happened — and the outbox steps between both by recording the intent first and relaying after.

§4

The shadow, pluggable behind one contract

shadow.ex / litestream.ex · verbatim surface

The off-box layer is named and pluggable, not built into the journal. EchoCache.Shadow (echo/apps/echo_cache/lib/echo_cache/shadow.ex) is a behaviour: @callback start_link/1; @callback restore/1 returning {:ok, :restored | :no_replica} | {:error, term}, restore-if-missing so an existing live file is never overwritten; @callback status/1; @callback stop/1. Two implementations honour it, and the supervisor and the restore path never know which one is wired.

ImplementationWhat it isThe contract
EchoCache.Litestreamthe off-box replicator: one server per journal directory, a rendered config naming each per-group database and its replica URL, the binary under a monitored Portrestore/1 rebuilds a group's file from its replica before the journal reopens — the first line of the node-death runbook; the shadow refuses a non-branded group
EchoCache.Shadow.Copythe laptop shadow: pure Elixir, VACUUM INTO snapshots to a local directory — zero binaries, zero credentials, runs anywhere Exqlite runsthe same four callbacks; :none always answers :no_replica and starts nothing

Wire by tuple: {EchoCache.Litestream, dir: ..., bucket: ...}, {EchoCache.Shadow.Copy, db: ..., dir: ...}, or :none. The milestone-A event store is the journal under a shadow; the move to per-instrument stream lanes is milestone B, the conn.1–conn.2 recorded dependency (docs/trading/trading.specs.md). The replay that is live state is B8.2.2 · Replay Equals Live.

§5

References

sources · related

Sources

Related