B4.4 · dive 1 — two memories, one file
Two memories in one file: the outbox and the last word.
One SQLite file per group, owned by one process, standing beside the bus. Inside it, two
tables: intents is the outbox at the writer's edge, applied is the lane's
last word per name at the applier's edge. The literature's answer to the dual-write trap is
Richardson's transactional outbox — record the intent in local durable storage first, relay it to the
broker after, and accept that the relay may deliver twice, so consumers "must be idempotent".
This series is unusually well-armed for that acceptance, and the law's wording is exact: "the bus gets
no WAL; the journal, which is not the bus, gets SQLite's."
Source: content/bcs4.4.md, quoting bcs_rung_4_4_check.out. This dive
reads the opening of the record — the self-check, the SQLite header, H1, the windows derive, and H2.
The transcript
bcs_rung_4_4_check.out · verbatim · this dive reads the opening: the header, H1, the windows derive, H202:15:17.826 [info] EchoData: contract self-check passed, codec=pure header: Valkey on 6390 | SQLite 3.46.0 via exqlite, WAL, synchronous=NORMAL | Elixir 1.14.0 OTP 25 | schedulers 1 H1 files ok -- one journal file per group, named by the group's branded id -- two groups, two files on disk -- a non-branded group is refused at the door, and the schema carries the two memories: intents (the outbox) and applied (the lane's last word per name) derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0} … (the memory, the loss, the compaction, and the price follow -- the full record is frozen on the module hub)
The opening [info] line and the SQLite header are part of the record: this
rung is the first cache rung to run under MIX_ENV=prod mix run, on the runtime's first
compiled dependency — exqlite, vendored as path deps under
runtimes/elixir/vendor/, the NIF compiled from the bundled SQLite amalgamation, no system
SQLite consulted.
Two memories in one file
content/bcs4.4.md · What · Where · HowThe intents table is the outbox: seq, the recorded job id, the
name, the version, an enqueued flag, nothing else — "the cargo law in a schema". The
applied table is the lane's last word per name: the newest version this group has ever
applied, surviving the table, the node, and the bus. The file per group is
journal-<group>.db under the declared directory, named by the group's branded id, and a
non-branded group is refused at the door — the same typed refusal every surface in this series makes
before it does any work.
The writer's edge is one verb, and the applier's edge is one handler — everything from 4.2's crash choreography still applies beneath it:
{:ok, _} =
EchoCache.Journal.start_link(
name: :limits_journal,
group: group_id, # the lane's PRT, one file per group
table: "limits",
dir: "/var/lib/echo/journals"
)
{:ok, job_id} = Journal.intend_and_enqueue(:limits_journal, conn, ast_id, version)
{:ok, _} =
EchoMQ.Consumer.start_link(
queue: Coherence.queue("limits"),
connector: [port: 6390],
handler: Journal.handler(:limits_journal, :limits)
)
Writers get the outbox in one verb, intend_and_enqueue/4, with the two seams
inside it drilled rather than assumed. Consumers swap one handler for another: Journal.handler/2
wires the lane through the memory. The journal itself is owned by one process — the single writer as a
storage discipline, which is what makes prepared-once statements trivially safe and SQLite's single-writer
nature a fit instead of a fight.
The seams, closed by what already exists
content/bcs4.4.md · What · DecisionsThe writer's flow is record, enqueue, mark — two seams. The rung drills both at once — a
never-enqueued intent and an enqueued-but-unmarked one — then calls replay, and the gate
closes on machinery this series already shipped: the bus's admission dedup from Part III and newer-wins
from 4.2. Step the drill below.
Static walk — 1) record: the intent lands in the outbox under a recorded job id
· 2) seam ①: a death before the enqueue leaves a pending intent · 3) replay
enqueues it — the never-enqueued intent replayed onto the bus · 4) enqueue: the job rides the
lane under its recorded id · 5) seam ②: a death after the enqueue but before the mark
leaves the bus holding the job · 6) replay reuses the recorded id and the bus answers
:duplicate at admission, counted · 7) both ended in the applied memory; once
coverage is total, replay is exactly %{replayed: 0, deduplicated: 0}.
Two decisions hold the seams open on purpose, then close them by replay.
mark_enqueued is a separate call: folding it into the enqueue would erase the seam the drill
exists to cover — "the outbox pattern's candor is that the seams are real", named, and closed by
replay rather than wished away. And replay reuses recorded job ids: minting fresh ids would defeat the
bus's admission dedup and deliver duplicates the consumer must absorb anyway; reusing the recorded id lets
the bus answer :duplicate for whatever survived, which the record counts in the open.
{:ok, %{replayed: n, deduplicated: m}} = Journal.replay(:limits_journal, conn)
Replay is Richardson's at-least-once relay, with the duplicate handling this series shipped two chapters ago — the admission dedup of B3.2 · Jobs Are Entities and the newer-wins comparison of B4.2 · Coherence by Mint Time. The next dive stages what the outbox is for: the bus dying with thirty obligations on it.
References
sources · relatedSources
- Richardson — Pattern: Transactional outboxthe outbox's contract and its crash seams
- SQLite — Write-Ahead Loggingthe commit path under both memories
- Litestream — How it worksthe off-box layer the boundary names and does not build
Related
- B4.4 — The Lane That Remembersthe module hub; the full rung in context
- B4 — EchoCachethe chapter landing; Part IV's arc
- B3.2 — Jobs Are Entitiesthe admission dedup that absorbs the replayed job id
- B4.2 — Coherence by Mint Timethe lane the journal stands beside
- B2.2 — Property Stores on ETSthe stores being cached
- EchoMQ — the coursethe protocol in rung-level depth
- Redis Patterns Appliedthe substrate under the bus
- Functional Programming in Elixirthe umbrella the runtimes live in