jonnify·redis /redis-patterns/caching/write-behind

R1.03 · Caching — module hub

Write-behind (write-back).

Maximize write throughput by writing only to Valkey and asynchronously syncing to the database later — trading immediate durability for performance. The cache acknowledges instantly; a background process synchronizes the changes, batching writes for efficiency.

Grounding: echo_cache/journal.ex (the outbox, the job lane, the SQLite WAL) · coherence.ex · content/bcs4.4.md — every figure verbatim from a committed record. Engine: Valkey.

One write, two paths. Select a path to read what runs on it and whether the caller waits.
a writeon the request Valkeyvalue + buffer flush workeron a timer database SET · ack now RPOP · batch
readout · Select a path. The write path returns when Valkey holds the change; the flush path reaches the database later, off the request path.
§1

How it works

write-behind.md.txt · How It Works

The pattern is three steps, and the database is reached on none of them. The application writes data to Valkey. Valkey acknowledges immediately — the client receives success. A background process periodically reads changed data from Valkey and writes it to the database. The store update happens “behind” the cache write, hence the name.

The application stores the new value and records that the key changed — a write buffer, built as a List with LPUSH or a dirty-set with SADD. The caller is acknowledged the moment both succeed. A separate background worker drains the buffer with RPOP on a timer, reads the current value for each changed key, and writes the batch to the database in one round trip. The source is reached far less often than the application writes, and never on the request’s critical path.

The figure above traces the split: the write path stores the value and returns; the flush path drains the buffer and reaches the database on the next tick. Nothing about the acknowledgement waits for the database.

The write returns when Valkey has it. The database is reached later, in batches, off the request path — throughput up front, the cost deferred.

§2

Redis commands used

write-behind.md.txt · journal.ex

The application writes to Valkey and nothing else. No database interaction occurs during these operations; the background sync process later reads these values and persists them.

SET    counter:page_views "15847293"
INCRBY counter:page_views 1

The write path adds one move to mark the key changed — LPUSH writebuf <key> onto a List buffer, or SADD dirty <key> onto a dirty-set — and the flush path drains it (RPOP, or SMEMBERS) and reads the current value before writing the source. On EchoCache the deferral is real code: the writer’s verb intend_and_enqueue/4 records the intent locally, hands the apply to the job lane over EchoMQ, and returns — the bus carries the apply off the write path.

§3

Advantages

write-behind.md.txt · Advantages
  • Extremely low write latency. Operations complete as soon as Valkey acknowledges — and, on EchoCache, after one local SQLite append — because no database round trip is on the request path.
  • High throughput. The database is protected from write storms. Many rapid updates collapse into a single database write during sync.
  • Write coalescing. If a counter is incremented 1000 times per second, only the final value needs to be written to the database, not 1000 individual increments. The third dive walks this step.
§4

The durability trade-off

write-behind.md.txt · journal.ex F3

This pattern trades durability for speed, and the cost is exact. Data loss risk: if the cache fails before synchronization, any writes since the last sync are lost — the cache becomes the system of record during the sync window. Eventual consistency: the database lags behind the cache, so a query straight to the database may return stale data.

EchoCache closes that window not with a cache-side append log but with the outbox: the intent is recorded in a SQLite WAL before the bus hears it, and replay/2 re-enqueues every intent not yet covered by the applied memory. The bus already deduplicates a re-enqueued job id, and newer-wins makes a re-applied version harmless. The plain-store dials still apply where the buffer is a raw List: persistence (AOF bounds loss to the last fsync; RDB to the last snapshot), shorter sync intervals, and replication. The choice between write-through and write-behind is a choice about where the acknowledgement happens.

Sync versus async. Pick a write strategy; the readout reports the cost over a fixed 100-write burst.
readout · Select a strategy. Write-through pays the database cost on the request; write-behind defers it and accepts a window.

Notes on ValkeySET key value PX <ms> sets the value and its TTL atomically — a separate EXPIRE after SET leaves an immortal key if the process dies between the two — valkey.io/commands/set.

Write-through returns after the database write and risks nothing on a cache loss. Write-behind returns after the cache write and risks the unflushed window — the trade the second dive measures.

§5

Ideal use cases — and when not to use

write-behind.md.txt · Use Cases

Write-behind excels where a small, bounded loss is acceptable in exchange for write throughput: counters and metrics (page views, likes, impressions), high-velocity session updates where database latency would be prohibitive, analytics ingestion at high speed, and coherence obligations on EchoCache — a changed risk limit must reach every cache copy, the job lane carries it at-least-once, and the journal makes the obligation survive a bus restart.

Never use write-behind for data where loss is unacceptable: financial transactions, order processing, user credentials or authentication data, or anything with regulatory compliance requirements.

§6

The sync process

write-behind.md.txt · journal.ex compact/1

The background synchronization typically works in four steps, and must handle failure so nothing is silently dropped. Scan for keys that have changed — using a dirty flag, a buffer List, or a timestamp. Read current values from the cache — so the source receives the latest state, never an intermediate one. Write to the database — batched, one round trip for many keys. Mark keys as synchronized — clear them from the buffer or dirty-set.

If a database write fails, the key must remain marked for retry — the buffer is the only copy until the source confirms. On EchoCache the equivalent is compact/1: an intent is retired only when its name carries an applied version at least as new — coverage, not acknowledgment, so the hot path pays no per-intent completion write. A burst of writes to one key collapses to a single source write: the dirty-set holds the key once however often it changed.

The sync process reads the latest value, writes one batch, and clears the dirty marks — and keeps a failed key dirty so a retry catches it.

§7

The three dives

the buffer → the window → the coalescing

The dives follow the arc the buffer → the window → the coalescing: how the asynchronous buffer absorbs a burst, the durability window the deferral opens and how to bound it, and how repeated writes to one key collapse into a single flush. Read them in order — the async buffer starts.

R1.03.1built

The async buffer

Write the value into Valkey, push the key onto a buffer with LPUSH, acknowledge at once, and drain to the database with RPOP on a timer.
R1.03.2built

The durability trade-off

The window where the cache holds the only copy, what a crash can lose, and how the flush interval and durable storage bound it. The journal’s measured price, verbatim.
R1.03.3built

Coalescing writes

Many updates to one key between flushes write the source once. A dirty-set keeps each key once; the flush reads the latest value.
write-behind

Acknowledge a write once the cache holds it; defer the source write to a background flush that batches and coalesces. Throughput rises; a bounded window of recent writes is at risk.

on EchoCache

Journal.intend_and_enqueue/4 records the intent in local SQLite and enqueues the apply on EchoMQ’s fair lanes; the apply happens off the write path, and replay/2 rebuilds the lane after a bus restart.

Three dives, one pattern: the buffer that absorbs the burst, the window that absorption opens, and the coalescing that makes a burst cost one write.

§8

The application — the job lane that carries the apply

coherence.ex · content/bcs4.4.md · verbatim

EchoCache.Journal is the write-behind path made durable: a per-group SQLite file standing beside the bus, holding the intents outbox (record → enqueue → mark) and the applied memory (the last version applied per name). The job lane is Coherence.enqueue/5 over EchoMQ’s queue ecc.coh.<table>; the cargo is the 29-byte coherence message id <> ":" <> version. The committed record prices the memory.

the application · the EchoMQ job lane, measured

The lane that remembers

The outbox is not free, and the price is committed, figure by figure, against the bare lane:

content/bcs4.4.md · EchoCache.Journal, the remembered lanefrozen
record-and-mark pair          143 us         ← the writer's edge
remembered-lane median        524 us         ← against the bare lane's 148
replay after a bus loss       exactly 30 uncovered intents re-enqueued in seq order
the trade                     3.5 times the latency buys an outbox, a last word
                              per name, and a replay that survives the bus

Notes on Valkeythe EchoMQ job lane rides at-least-once over Valkey; the journal’s SQLite WAL runs at synchronous=NORMAL, where the checkpoint is the only sync barrier — the durability-for-throughput trade the lane accepts — valkey.io/topics/pubsub (the broadcast contrast).

§9

References

sources · related

Sources

Related in this course