B8.2 · The Log and the Ledger — the trading corpus, docs/trading/
Two memories: the log, and the ledger.
A trading platform keeps two kinds of memory, and they belong in two places. The stream
— the append-only log of every fill — is the source of truth for unsettled state, and recovery from it
is replay, not a feature. The ledger — settled money, posted double-entry — is the regulated record of
settled positions. The log layer is the as-built EchoCache.Journal under a pluggable
EchoCache.Shadow, quoted from its committed record; the regulated Trading.Ledger is
a PROPOSED design object.
The journal is real Elixir in the live umbrella
(echo/apps/echo_cache/lib/echo_cache/journal.ex), on the committed B4.4 record
(content/bcs4.4.md, bcs_rung_4_4_check.out, PASS 6/6) — every journal
figure is verbatim from there. The Trading.Ledger / Exchange.Projection modules are
the Operator's design (docs/trading/), status PROPOSED: taught as design, never as a measured
platform number.
The three dives
fixed by the chapter spec · the trading corpusThe module is one memory layer in three dives — the journal beside the bus and its pluggable
shadow, the replay that is live state, and the compaction and the regulated ledger. The journal each
dive quotes is as-built and committed; the Trading.Ledger and Exchange.Projection
are PROPOSED design objects, taught as design and never as a measured result.
The Journal and the Shadow
One EchoCache.Journal per book, named by its branded id; the two memories; both crash
seams closed by what already exists — once coverage is total, replay is exactly %{replayed: 0,
deduplicated: 0}; the pluggable EchoCache.Shadow (Litestream off-box, Copy on a
laptop).
Replay Equals Live
State is the fold of the log. The journal remembers across a full restart
(:remembered_stale from the journal alone, no cache row touched); the loss drill —
exactly 30 uncovered intents re-enqueued in seq order, 50 of 50; recovery is
replay, re-gated per book.
The Double-Entry Ledger
Compaction — the outbox empties, the memory does not; the PROPOSED
Trading.Ledger — settled money double-entry in Postgres, one Ecto.Multi per
posting; the H6 price pair 524 µs against the bare lane's 148 µs.
The module hands forward inside the capstone: B8.3 · Strategies as
Deciders — strategies emitting intents over the same log; then B8.4 · Fan-Out and the
Scale-Out. Those modules and the Trading.Ledger / Exchange.Projection
objects this module names are PROPOSED design, not routes.
Two memories, one file
bcs_rung_4_4_check.out · the schemaThe journal carries two memories, and the schema is the cargo law. The intents table is Richardson's transactional outbox — record the intent before the bus hears it, mark it after. The applied table is the lane's last word per name — the newest version this lane has ever applied, surviving the table, the node, and the bus. The committed gate states it exactly.
| Memory | Carries | Role (as-built — bcs_rung_4_4_check.out) |
|---|---|---|
| intents | seq, recorded job_id, name_id, version, enqueued flag, recorded_at | the transactional outbox: record before the enqueue, mark after; replay re-enqueues every uncovered intent under its recorded job id |
| applied | name_id (key), the newest version, its seq | the lane's last word per name; lives in the file, so it survives the table, the node, and the bus; the dedup floor and the retention rule |
The as-built memory — the journal, committed
quoting bcs_rung_4_4_check.outThe module invents nothing under the memory layer. EchoCache.Journal is real
Elixir in the live umbrella (echo/apps/echo_cache/lib/echo_cache/journal.ex), one per group,
owned by one process. Its committed behavior is the whole six-gate record — the schema, both crash seams, the
cross-restart memory, the loss drill, compaction, and the price — quoted verbatim. These are the as-built
journal's committed numbers, never the trading platform's own.
02:15:17.826 [info] EchoData: contract self-check passed, codec=pure header: Valkey on 6390 | SQLite 3.46.0 via exqlite, WAL, synchronous=NORMAL | Elixir 1.14.0 OTP 25 | schedulers 1 H1 files ok -- one journal file per group, named by the group's branded id -- two groups, two files on disk -- a non-branded group is refused at the door, and the schema carries the two memories: intents (the outbox) and applied (the lane's last word per name) derive (windows): the writer's flow is record, enqueue, mark -- two crash seams; a death before the enqueue leaves a pending intent that replay enqueues; a death after the enqueue but before the mark leaves the bus holding the job, and replay's reuse of the recorded job id lets the bus's own admission dedup absorb it; full coverage by the applied memory makes replay a no-op H2 windows ok -- both crash seams closed by machinery that already exists: the never-enqueued intent replayed onto the bus, the enqueued-but-unmarked one answered :duplicate at admission and was counted, both ended in the applied memory -- and once coverage is total, replay is exactly %{replayed: 0, deduplicated: 0} derive (memory): the applied table lives in the file, so it survives the table, the node, and the bus; after a full restart with an empty L1, a replayed old version must answer :remembered_stale from the journal alone -- no cache row consulted, none created -- while a genuinely newer version passes through and updates the memory H3 memory ok -- the journal remembered v5 across a full stop of table and journal; the replayed old version answered :remembered_stale without touching the cache -- the table's verdict counters did not move and no row appeared -- and the genuinely newer v6 passed through and became the new last word derive (loss): D-2 keeps the bus volatile, so a bus restart erases queued coherence jobs; 50 intents recorded and enqueued, 20 applied before the loss, the lane's queue keys flushed -- replay must re-enqueue exactly the 30 uncovered intents in seq order, and a consumer must drain them to a remembered count of 50 H4 loss ok -- the bus restart erased the queue and the journal replayed the lane back: exactly 30 uncovered intents re-enqueued in seq order under their recorded job ids, the consumer drained them, and the applied memory closed at 50 of 50 names holding their final versions derive (compaction): an intent is retired when its name carries an applied version at least as new -- coverage, not acknowledgment -- so after H4 all 50 intents are deletable, the applied memory keeps all 50 names, replay finds nothing, and a reopen still remembers H5 compaction ok -- all 50 intents retired by coverage in one pass, the applied memory kept its 50 names, replay over the compacted journal found nothing to do, and a fresh open of the same file still answers the last word -- the outbox empties, the memory does not derive (price): on prepared-once statements -- the single writer's privilege, with bind resetting the statement -- the writer's pair is two WAL commits and one cached rowid read at synchronous=NORMAL, so expect between 20 and 250 us on this disk; the remembered lane's end-to-end median should land between 200 us and 2 ms against the bare lane's committed 148 us: dearer, bounded, and the chapter's reason the journal is declared per group rather than assumed H6 price ok -- the memory's price on this disk: 143 us per record-and-mark pair at the writer's edge, and a remembered lane end-to-end median of 524 us against the bare lane's committed 148 us -- 3.5 times the latency buys an outbox, a last word per name, and a replay that survives the bus PASS 6/6
The platform's own numbers — the settlement latency, the posting throughput, the
fill counts — do not exist yet, and so they appear nowhere on these pages. The first such number lands
when the first rung's harness runs (docs/trading/trading.roadmap.md, TRD.3).
The shadow, pluggable — and the regulated ledger, PROPOSED
shadow.ex / litestream.ex · docs/trading/trading.specs.mdThe off-box layer is named and pluggable, not built into the journal.
EchoCache.Shadow (echo/apps/echo_cache/lib/echo_cache/shadow.ex) is a behaviour —
one contract, start_link/1, restore/1 returning
{:ok, :restored | :no_replica}, status/1, stop/1 — with two
implementations: EchoCache.Litestream (litestream.ex) replicating each per-group
journal to S3-compatible object storage and restoring before the journal reopens, and
EchoCache.Shadow.Copy snapshotting to a local directory on a development laptop. The supervisor
and the restore path never know which one is wired — the same contract either way, and the kind law
holds at the shadow's door.
The settled side is the design's Trading.Ledger (PROPOSED,
docs/trading/trading.specs.md): settled money posts double-entry in Postgres, branded id columns
under the SQL canon, every posting one Ecto.Multi — a single all-or-nothing transaction per
posting. The directional decision is named in the spec: the stream is the source of truth for unsettled
state, the Postgres ledger the regulated record of settled positions. Exchange.Projection
(also PROPOSED) folds the log idempotently into Tables. Both are taught as design; neither has a rung yet, and
so neither claims a number.
The doors
the sibling coursesEchoMQ — the protocol, in depth
The bus the log rides, and the per-instrument stream lanes the log moves to at milestone B, taught rung by rung.
/redis-patternsRedis Patterns Applied
The substrate under the memory: the transactional outbox, the durable last word, the single-writer move.
/elixirFunctional Programming in Elixir
The umbrella the journal and shadow live in, and the functional fold replay is.
References
sources · relatedSources
- Richardson — Pattern: Transactional outboxrecord the intent locally first, relay to the broker after; the relay may deliver twice, so consumers must be idempotent — the journal's intents outbox
- SQLite — Write-Ahead Loggingthe WAL commit path; at synchronous NORMAL the checkpoint is the sole sync barrier — the journal's speed and its stated power-loss trade
- Litestream — How it worksa separate process copies WAL pages to replicas and restores by snapshot plus replay — the off-box EchoCache.Litestream shadow
Related
- B8 — The Trading Systemthe chapter landing; the capstone's arc
- B8.1 — The Enginethe engine whose events land in this log
- B4.4 — The Lane That Remembersthe as-built journal quoted here, in its home chapter
- B4 — EchoCachethe cache the journal's memory outlives
- B3 — The Busthe work lanes the journal's outbox feeds
- B2 — The Elixir BCS Corethe functional core replay is the fold of
- EchoMQ — the coursethe bus protocol in rung-level depth
- Redis Patterns Appliedthe durable-memory substrate
- Functional Programming in Elixirthe umbrella the journal lives in