B8.2 · dive 1 — the journal and the shadow
Durability beside the bus, not inside it.
A trading platform's log answers one obligation: every fill must survive the bus. The bus is
volatile by decision D-2 — a restart erases its queued jobs — so durability lives beside the bus. The as-built
EchoCache.Journal is that durability: a per-group SQLite file holding two memories, the
outbox of intents at the writer's edge and the last applied version per name at the applier's
edge. The committed gate fixes the schema: one journal file per group, named by the group's branded id ...
the schema carries the two memories.
Source: the journal is real Elixir at
echo/apps/echo_cache/lib/echo_cache/journal.ex, taught in content/bcs4.4.md, its
record bcs_rung_4_4_check.out closing PASS 6/6. The book that owns one per
instrument, Exchange.Book, is PROPOSED (docs/trading/) — taught as design,
never as a measured number.
The two memories, in one file
echo/apps/echo_cache/lib/echo_cache/journal.ex · verbatim surfaceThe intents table is Richardson's transactional outbox: seq, the
recorded job_id, the name_id, the version, an enqueued
flag, a recorded_at — the cargo law in a schema, nothing else. The applied table is
the lane's last word: name_id primary key, the newest version, its seq.
The applied table lives in the file, so it survives the table, the node, and the bus.
The as-built surface, quoted from source — two writer edges and an outbox verb:
# the writer's first edge: record the intent before the bus hears it record(j, job_id, name_id, version) # the writer's second edge: the bus accepted this intent mark_enqueued(j, job_id) # group commit: one WAL append amortized across a batch record_many(j, triples) # the outbox in one verb: mint a job id, record, enqueue, mark intend_and_enqueue(j, conn, name_id, version) # a consumer handler wiring the job lane through the journal's memory handler(j, table) replay(j, conn) # recovery beside the bus compact(j) # retire every intent whose name carries an applied version at least as new
Two doors, one law: a non-branded group is refused (a non-branded group is refused at
the door). One journal file per group, one owner process per journal — the single writer of Chapter
4.3 as a storage discipline, which is what makes the five hot statements prepared once at journal start and
rebound per call trivially safe.
Both crash seams, closed by what already exists
bcs_rung_4_4_check.out · H2 · verbatimThe writer's flow is record, enqueue, mark — two crash seams. A death before the enqueue leaves a pending intent that replay enqueues. A death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it. Neither seam needs new machinery: the bus already deduplicates a job id at admission, and newer-wins already makes a re-applied version a no-op.
:duplicate at admission. The verdict is the
H2 record's own.record. Advance it through enqueue and mark, or choose a death
point and run replay. Coverage by the applied memory makes a fully-marked intent's replay a no-op.derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0} PASS 6/6
The journal's WAL is beside the bus, not inside it
content/bcs4.4.md · D-2The law's wording is exact: the bus gets no WAL; the journal, which is not the bus, gets
SQLite's. D-2 stands untouched — the bus stays volatile, restart semantics stay replay, and durability lives
in a file the bus has never heard of. The journal opens PRAGMA journal_mode=WAL,
synchronous=NORMAL, busy_timeout=5000 — the last because the shadow takes brief
write locks at checkpoint and the journal waits rather than errors.
mark_enqueued is a separate call on purpose: folding it into the enqueue would
erase the seam the drill exists to cover. The outbox pattern's candor is that the seams are real, named, and
closed by replay rather than wished away. The dual-write trap waits on either side of the obvious fix — write
the store then enqueue and a crash loses the message; enqueue then write and a crash announces a write that
never happened — and the outbox steps between both by recording the intent first and relaying after.
The shadow, pluggable behind one contract
shadow.ex / litestream.ex · verbatim surfaceThe off-box layer is named and pluggable, not built into the journal.
EchoCache.Shadow (echo/apps/echo_cache/lib/echo_cache/shadow.ex) is a behaviour:
@callback start_link/1; @callback restore/1 returning
{:ok, :restored | :no_replica} | {:error, term}, restore-if-missing so an existing live file is
never overwritten; @callback status/1; @callback stop/1. Two implementations honour
it, and the supervisor and the restore path never know which one is wired.
| Implementation | What it is | The contract |
|---|---|---|
| EchoCache.Litestream | the off-box replicator: one server per journal directory, a rendered config naming each per-group database and its replica URL, the binary under a monitored Port | restore/1 rebuilds a group's file from its replica before the journal reopens — the first line of the node-death runbook; the shadow refuses a non-branded group |
| EchoCache.Shadow.Copy | the laptop shadow: pure Elixir, VACUUM INTO snapshots to a local directory — zero binaries, zero credentials, runs anywhere Exqlite runs | the same four callbacks; :none always answers :no_replica and starts nothing |
Wire by tuple: {EchoCache.Litestream, dir: ..., bucket: ...},
{EchoCache.Shadow.Copy, db: ..., dir: ...}, or :none. The milestone-A event store is
the journal under a shadow; the move to per-instrument stream lanes is milestone B, the conn.1–conn.2 recorded
dependency (docs/trading/trading.specs.md). The replay that is live state is
B8.2.2 · Replay Equals Live.
References
sources · relatedSources
- Richardson — Pattern: Transactional outboxrecord the intent locally first, relay after; the relay may deliver twice — the journal's intents outbox
- SQLite — Write-Ahead Loggingthe WAL commit path; at synchronous NORMAL the checkpoint is the sole sync barrier — the journal's journal_mode=WAL and synchronous=NORMAL
- Litestream — How it worksa separate process copies WAL pages to replicas and restores by snapshot plus replay — the EchoCache.Litestream shadow
Related
- B8.2 — The Log and the Ledgerthe module hub; the full journal record in context
- B4.4 — The Lane That Remembersthe as-built journal this dive quotes, in its home chapter
- B4 — EchoCachethe chapter the journal lives in
- B3 — The Busthe work lanes the outbox feeds
- Redis Patterns Appliedthe transactional-outbox substrate
- Functional Programming in Elixirthe umbrella the journal and shadow live in