jonnify·bcs /bcs/trading/log-and-ledger

B8.2 · The Log and the Ledger — the trading corpus, docs/trading/

Two memories: the log, and the ledger.

A trading platform keeps two kinds of memory, and they belong in two places. The stream — the append-only log of every fill — is the source of truth for unsettled state, and recovery from it is replay, not a feature. The ledger — settled money, posted double-entry — is the regulated record of settled positions. The log layer is the as-built EchoCache.Journal under a pluggable EchoCache.Shadow, quoted from its committed record; the regulated Trading.Ledger is a PROPOSED design object.

The journal is real Elixir in the live umbrella (echo/apps/echo_cache/lib/echo_cache/journal.ex), on the committed B4.4 record (content/bcs4.4.md, bcs_rung_4_4_check.out, PASS 6/6) — every journal figure is verbatim from there. The Trading.Ledger / Exchange.Projection modules are the Operator's design (docs/trading/), status PROPOSED: taught as design, never as a measured platform number.

The two memories in one file. Select a memory to read its columns and its role over a fixed two-name example — the schema's own shape, computed from the record.
out intents · the outbox last applied · the last word
readout · Select a memory to read its columns and its role. The intents table is the transactional outbox; the applied table is the lane's last word per name — and the second outlives the table, the node, and the bus.
§1

The three dives

fixed by the chapter spec · the trading corpus

The module is one memory layer in three dives — the journal beside the bus and its pluggable shadow, the replay that is live state, and the compaction and the regulated ledger. The journal each dive quotes is as-built and committed; the Trading.Ledger and Exchange.Projection are PROPOSED design objects, taught as design and never as a measured result.

The module hands forward inside the capstone: B8.3 · Strategies as Deciders — strategies emitting intents over the same log; then B8.4 · Fan-Out and the Scale-Out. Those modules and the Trading.Ledger / Exchange.Projection objects this module names are PROPOSED design, not routes.

§2

Two memories, one file

bcs_rung_4_4_check.out · the schema

The journal carries two memories, and the schema is the cargo law. The intents table is Richardson's transactional outbox — record the intent before the bus hears it, mark it after. The applied table is the lane's last word per name — the newest version this lane has ever applied, surviving the table, the node, and the bus. The committed gate states it exactly.

MemoryCarriesRole (as-built — bcs_rung_4_4_check.out)
intentsseq, recorded job_id, name_id, version, enqueued flag, recorded_atthe transactional outbox: record before the enqueue, mark after; replay re-enqueues every uncovered intent under its recorded job id
appliedname_id (key), the newest version, its seqthe lane's last word per name; lives in the file, so it survives the table, the node, and the bus; the dedup floor and the retention rule
One journal file per group. Named by the group's branded id; one owner process per journal — the single writer of Chapter 4.3 as a storage discipline. A non-branded group is refused at the door. The bus gets no WAL; the journal, which is not the bus, gets SQLite's. Durability lives in a file the bus has never heard of, which keeps D-2 untouched: the bus stays volatile, recovery stays replay.
§3

The as-built memory — the journal, committed

quoting bcs_rung_4_4_check.out

The module invents nothing under the memory layer. EchoCache.Journal is real Elixir in the live umbrella (echo/apps/echo_cache/lib/echo_cache/journal.ex), one per group, owned by one process. Its committed behavior is the whole six-gate record — the schema, both crash seams, the cross-restart memory, the loss drill, compaction, and the price — quoted verbatim. These are the as-built journal's committed numbers, never the trading platform's own.

content/echo_data/runtimes/elixir/bcs_rung_4_4_check.out · the memory the ledger stands onfrozen
02:15:17.826 [info] EchoData: contract self-check passed, codec=pure
header: Valkey on 6390 | SQLite 3.46.0 via exqlite, WAL, synchronous=NORMAL | Elixir 1.14.0 OTP 25 | schedulers 1
H1 files ok -- one journal file per group, named by the group's branded id -- two groups, two files on disk -- a non-branded group is refused at the door, and the schema carries the two memories: intents (the outbox) and applied (the lane's last word per name)
derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op
H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0}
derive (memory): the applied table lives in the file, so it survives the table, the node, and the bus; after a full restart with an empty L1, a replayed old version must answer :remembered_stale from the journal alone -- no cache row consulted, none created -- while a genuinely newer version passes through and updates the memory
H3 memory ok -- the journal remembered v5 across a full stop of table and journal; the replayed old version answered :remembered_stale without touching the cache -- the table's verdict counters did not move and no row appeared -- and the genuinely newer v6 passed through and became the new last word
derive (loss): D-2 keeps the bus volatile, so a bus restart erases queued coherence jobs; 50 intents recorded and enqueued, 20 applied before the loss, the lane's queue keys flushed -- replay must re-enqueue exactly the 30 uncovered intents in seq order, and a consumer must drain them to a remembered count of 50
H4 loss ok -- the bus restart erased the queue and the journal replayed the lane back: exactly 30 uncovered intents re-enqueued in seq order under their recorded job ids, the consumer drained them, and the applied memory closed at 50 of 50 names holding their final versions
derive (compaction): an intent is retired when its name carries an applied version at least as new -- coverage, not acknowledgment -- so after H4 all 50 intents are deletable, the applied memory keeps all 50 names, replay finds nothing, and a reopen still remembers
H5 compaction ok -- all 50 intents retired by coverage in one pass, the applied memory kept its 50 names, replay over the compacted journal found nothing to do, and a fresh open of the same file still answers the last word -- the outbox empties, the memory does not
derive (price): on prepared-once statements -- the single writer's privilege, with bind resetting the statement -- the writer's pair is two WAL commits and one cached rowid read at synchronous=NORMAL, so expect between 20 and 250 us on this disk; the remembered lane's end-to-end median should land between 200 us and 2 ms against the bare lane's committed 148 us: dearer, bounded, and the chapter's reason the journal is declared per group rather than assumed
H6 price ok -- the memory's price on this disk: 143 us per record-and-mark pair at the writer's edge, and a remembered lane end-to-end median of 524 us against the bare lane's committed 148 us -- 3.5 times the latency buys an outbox, a last word per name, and a replay that survives the bus
PASS 6/6

The platform's own numbers — the settlement latency, the posting throughput, the fill counts — do not exist yet, and so they appear nowhere on these pages. The first such number lands when the first rung's harness runs (docs/trading/trading.roadmap.md, TRD.3).

§4

The shadow, pluggable — and the regulated ledger, PROPOSED

shadow.ex / litestream.ex · docs/trading/trading.specs.md

The off-box layer is named and pluggable, not built into the journal. EchoCache.Shadow (echo/apps/echo_cache/lib/echo_cache/shadow.ex) is a behaviour — one contract, start_link/1, restore/1 returning {:ok, :restored | :no_replica}, status/1, stop/1 — with two implementations: EchoCache.Litestream (litestream.ex) replicating each per-group journal to S3-compatible object storage and restoring before the journal reopens, and EchoCache.Shadow.Copy snapshotting to a local directory on a development laptop. The supervisor and the restore path never know which one is wired — the same contract either way, and the kind law holds at the shadow's door.

The settled side is the design's Trading.Ledger (PROPOSED, docs/trading/trading.specs.md): settled money posts double-entry in Postgres, branded id columns under the SQL canon, every posting one Ecto.Multi — a single all-or-nothing transaction per posting. The directional decision is named in the spec: the stream is the source of truth for unsettled state, the Postgres ledger the regulated record of settled positions. Exchange.Projection (also PROPOSED) folds the log idempotently into Tables. Both are taught as design; neither has a rung yet, and so neither claims a number.

§5

The doors

the sibling courses
§6

References

sources · related

Sources

  • Richardson — Pattern: Transactional outboxrecord the intent locally first, relay to the broker after; the relay may deliver twice, so consumers must be idempotent — the journal's intents outbox
  • SQLite — Write-Ahead Loggingthe WAL commit path; at synchronous NORMAL the checkpoint is the sole sync barrier — the journal's speed and its stated power-loss trade
  • Litestream — How it worksa separate process copies WAL pages to replicas and restores by snapshot plus replay — the off-box EchoCache.Litestream shadow

Related