A jonnify course
EchoMQ In Depth
One Valkey, one wire you own — a queue, a bus, and a cache, from a single protocol.
EchoMQ is one Valkey-native job system you own end to end. Below the language line sit the keys and
the Lua scripts — the protocol; above it stand the three surfaces you reach for every day: the
Queue that distributes work, the Bus that broadcasts signals and keeps a
replayable event log, and the Cache that serves reads.
This course teaches their internals in depth, and Elixir-canonical — the real keys, the real Lua,
and the real module functions, lifted onto the page and annotated — and what they mean for every system in the
BCS family that rides them. Because the protocol lives below the language line, any runtime that
speaks the keys and the scripts is a peer, so EchoMQ is supported in and ported to others. It is the course on the
far side of every “→ EchoMQ” door in
Redis Patterns Applied.
One system, three pillars
EchoMQ is not a library you bolt onto an app — it is a protocol you own. The keys and the Lua scripts that run against Valkey are the system; everything a caller touches sits above them. Draw the line where the agreement lives: below it, the keys and the atomic scripts, fixed and shared; above it, the language you happen to write in. Push the agreement below that line and three surfaces fall out of one wire — the Queue (distribute work: enqueue, claim, retry, complete), the Bus (broadcast signals: pub/sub events and a retained, replayable log), and the Cache (serve reads: an L1 ETS / L2 Valkey near-cache kept coherent on the Bus).
Why Valkey. EchoMQ is backed by Valkey — the BSD-licensed, foundation-governed fork with Redis-semantics scripting, AOF durability, and hash-field TTL. It is the neutral substrate of record, the store every durability and coherence claim is phrased against, and it outlives any one vendor's licensing mood.
The course map · pick a chapter
One protocol below the line; the Queue, the Bus, and the Cache above it — open the chapter you need.
What's inside · the map
The whole system, chapter by chapter. Read the foundation first — the Overview, then the Protocol that the three pillars share. Then take the pillar you need: the Queue, the Bus, the Cache. The Proof chapter shows the system holds — conformance, telemetry, the benchmark. Each chapter closes with a hands-on workshop.
The system & its protocol
What EchoMQ is, and the wire every pillar stands on — the owned keyspace, the record hash, the Lua layer, and the immutability that lets one protocol serve three surfaces. Taught in depth on the real code.
Overview
Open chapter →One system, three pillars; the protocol below the line; the Redis-Patterns door and what the pillars mean for the BCS family.
← redis-patterns
The three pillars
The Queue, the Bus, the Cache — one system, three surfaces over one wire.
The protocol below the line
Keys + Lua shared and fixed; the language above. Why that line makes it polyglot and coherent.
The door & the BCS family
The bidirectional Redis-Patterns door, and what EchoMQ means for the systems built on it.
The Protocol
Open chapter →The substrate the three pillars share: the owned keyspace, the record hash, the Lua layer, immutability and branded ids.
← redis-patterns R0.3 · R2
The owned keyspace
emq:{q}: with a hashtag per queue, the {emq}: reserve, and the slot the hashtag pins.
The record hash
The job row: the HASH, the fields, and what each holds — part of the protocol.
The Lua layer & EVALSHA
The scripts are the protocol: declared keys, load-once, run by SHA, the atomic move.
Immutability & branded ids
The immutable line below the language; the 14-byte branded-id gate and the version fence.
Workshop
Decode a real key, then trace one EVALSHA back to its Lua source.
What EchoMQ does
The three surfaces over the one wire — distribute work, broadcast signals, serve reads. Each applies a Redis pattern you can follow back to Redis Patterns Applied, and each is grounded in the real Elixir code.
The Queue
Open chapter →Distribute work: the lifecycle, jobs · lanes · the consumer, fair groups, batches, lifecycle controls, and flows.
← redis-patterns R3 · R4 · R6
The lifecycle
The state machine: enqueue → claim → complete or retry; a script per transition.
Jobs, lanes & the consumer
The producer surface, fair grouped lanes, and the supervised consumer loop.
Batches
Bulk in one flush: enqueue-many, bulk flows, and batch lease-extension.
Lifecycle controls
Scheduling and recurrence, cooperative cancel and checkpoints, and the operator plane.
Flows
Orchestration: parent and child jobs, fan-in on completion, cross-queue delivery.
Workshop
Run a job through every state; compose a flow and watch the fan-in complete.
The Bus
Broadcast signals: pub/sub events, the retained replayable event log, consumer groups, time-travel and the archive.
on the build front← redis-patterns R5 · R6
Pub/sub events
The event seam: subscribe to a queue's lifecycle, publish a signal, decoupled one-to-many.
The event log
The retained, replayable stream; append is mint order; read at your own offset.
Consumer groups
At-least-once delivery, crash re-delivery, and the polyglot reader seam.
Time-travel & the archive
Retention as policy; the archive under the cache's shadow; reads by mint instant.
Workshop
Publish and subscribe across nodes; replay a window of the log by mint time.
The Cache
Serve reads: a two-layer cache-aside near-cache, single-flight filled, jittered-TTL swept, coherent on the Bus.
on the build front← redis-patterns R1 · R5
Cache-aside, two layers
fetch over L1 ETS at memory speed and L2 Valkey on a miss; the branded key.
Single-flight & jittered TTL
One loader for a thundering herd; cohorts that never expire in step.
Coherence — newer wins on the Bus
Version-guarded fills; invalidation rides the Bus, newer mint wins.
Workshop
Hit at ETS speed; invalidate from another node; survive a herd with one fill.
The system holds
How the wire is kept honest across every runtime — the black-box conformance suite, the telemetry catalog, and the benchmark gate.
The Proof
Conformance, telemetry & the benchmark gate — the evidence the protocol is correct, observable, and fast.
on the build front← redis-patterns R8
The conformance suite
The black-box scenarios that keep the owned wire honest across runtimes.
Telemetry & tracing
The event catalog and the OpenTelemetry / W3C mapping.
The benchmark gate
The honest throughput table, with the candid trade recorded.
Workshop
Run the suite, read the table, operate the fleet.
Canonical in Elixir, polyglot by design
This course teaches EchoMQ through its canonical Elixir implementation — the real modules and the real Lua, lifted onto the page and annotated. That is where the depth is. But the depth is portable: because the protocol is the keys and the scripts, not the library, any runtime that speaks them is a peer. EchoMQ is supported in and ported to other runtimes; what you learn here — the keyspace, the record, the scripts, the lifecycle — is the same wire they speak. The pillars are taught once, in Elixir, and they hold wherever the protocol runs.
Learn the wire once, in Elixir; it is the same wire every peer runtime speaks.
Why the BCS architecture builds on EchoMQ
EchoMQ is not a library bolted onto an app — it is the queue, the bus, and the cache of the Branded Component System. The BCS course teaches the architecture: the 14-byte branded snowflake, the grammar-total keyspace law, fair lanes, and a coherent cache-aside near-cache. EchoMQ is that architecture, built — the bus (jobs, lanes, the consumer) and the cache (the L1/L2 near-cache, kept coherent on the bus) are the same modules this course opens up.
So a system built on the BCS architecture inherits its queue, its cache, and its coherence from EchoMQ for free. The worked example is the Exchange Platform, the BCS capstone: a matching engine whose settlement jobs, market-data claims, and position cache all ride EchoMQ. Read this course to learn the bus; read the BCS course to see the architecture it completes.
Learn EchoMQ here; see where it lands — the whole branded architecture — in the BCS course.
References
Sources
- Redis — Documentation — the data structures and commands the protocol is built from.
- Redis — RPOPLPUSH — the reliable wait→active move at the heart of the lifecycle.
- llmstxt.org — The llms.txt convention — the machine-readable map format the course follows.
Related in this course
- Overview — one system, three pillars, and the door.
- The Protocol — the owned keyspace, the Lua layer, immutability.
- Redis Patterns Applied — the near side of the door; the pattern source.
- The Branded Component System — the architecture EchoMQ completes.