jonnifyechomq
EchoMQ / Overview / The three pillars

Overview · dive 1 · what

The three pillars

One wire, three surfaces. The Queue distributes work, the Bus broadcasts signals, the Cache serves reads.

EchoMQ is one Valkey-native job system you own, canonical in Elixir. Above the one wire it owns stand three pillars, and each answers a different question about moving data between the parts of a system: the Queue hands one job to one worker, the Bus announces one fact to many listeners, and the Cache keeps one hot value close. This dive names the three surfaces and the real function that opens each one; the deep mechanism is the pillar chapters on the course map. Pick a pillar on the right to read its job and its shape.

The three surfaces Three shapes Which pillar fits References

Three pillars over one wire · pick a pillar

The Queue distribute work one job one worker point to point The Bus broadcast signals one event many listeners one to many The Cache serve reads one value read often read through one owned wire the keyspace + the Lua layer

Three surfaces, one wire

EchoMQ owns one Valkey keyspace and one set of atomic Lua scripts. Above that line sit three product surfaces, and the difference between them is the question each one answers. The Queue distributes work: a task is handed to exactly one worker, runs once, and is marked done. The Bus broadcasts signals: a fact is announced, and every interested listener hears it. The Cache serves reads: a hot value is held close, so a read does not pay the full cost twice. Three surfaces, three jobs — one substrate underneath.

Each surface is a real module. The point here is which function is the door into each pillar; the mechanism behind the door is taught in depth in the pillar chapters.

The Queue — distribute work

A job is admitted with EchoMQ.Jobs.enqueue/4, claimed by a worker with EchoMQ.Jobs.claim/3, and the loop that owns the claim rhythm is EchoMQ.Consumer. Admission is one atomic step on the server: the Elixir verb passes the job key and the pending set as the two declared keys, and the script does the rest.

# echo/apps/echo_mq — EchoMQ.Jobs.enqueue/4
# Admit a job in one atomic step. The verb passes the job key and the
# pending set as the two declared KEYS; the named @enqueue script does the rest.
def enqueue(conn, queue, job_id, payload) when is_binary(job_id) and is_binary(payload) do
  keys = [Keyspace.job_key(queue, job_id), Keyspace.queue_key(queue, "pending")]

  case Connector.eval(conn, @enqueue, keys, [job_id, payload]) do
    {:ok, 1} -> {:ok, :enqueued}   # admitted
    {:ok, 0} -> {:ok, :duplicate}  # already present — idempotent
    {:error, {:server, "EMQKIND" <> _}} -> {:error, :kind}  # not a JOB-namespaced id
    other -> other
  end
end

The admission rule itself lives in one named Lua handle — EchoMQ.Jobs @enqueue — run on the server so the kind check, the duplicate refusal, the row write, and the pending insertion happen as one indivisible step.

-- EchoMQ.Jobs @enqueue — the atomic admission script
-- KEYS[1] = the job key   KEYS[2] = the pending set   ARGV[1] = id   ARGV[2] = payload
if string.sub(ARGV[1], 1, 3) ~= 'JOB' then          -- the branded-id gate: ids are JOB-namespaced
  return redis.error_reply('EMQKIND job id must be JOB-namespaced')
end
if redis.call('EXISTS', KEYS[1]) == 1 then          -- already admitted? refuse, idempotently
  return 0
end
redis.call('HSET', KEYS[1], 'state', 'pending', 'attempts', '0', 'payload', ARGV[2])  -- write the row
redis.call('ZADD', KEYS[2], 0, ARGV[1])               -- same score: byte order is mint order
return 1

The Bus — broadcast signals

A listener subscribes with EchoMQ.Events.subscribe/2; a lifecycle fact is announced with EchoMQ.Events.publish/5 on the per-queue channel EchoMQ.Events.channel/1 (emq:{q}:events). One event reaches every subscribed listener — the broadcast shape. Beyond the live broadcast, the Bus keeps a retained, replayable event log: append order is mint order, a reader can start at an offset and read forward, and a consumer group recovers what it missed.

# echo/apps/echo_mq — EchoMQ.Events.publish/5
# Announce a lifecycle fact to every listener on the per-queue channel.
def publish(conn, queue, event, job_id, extra \\ []) do
  _ = Keyspace.job_key(queue, job_id)        # gate the id at the key builder before the wire
  payload = encode_event(event, job_id, extra)

  case Connector.command(conn, ["PUBLISH", channel(queue), payload]) do
    {:ok, _n} -> :ok                          # delivered to live subscribers
    other -> other
  end
end

The Bus has two faces: the live pub/sub broadcast above, and a retained event log a consumer can replay from an offset. The event log — mint-ordered append, read-at-offset, consumer-group recovery — is taught in depth in the Bus pillar chapter.

The Cache — serve reads

A hot value is read with EchoCache.Table.fetch/3: a hit returns from L1 (an ETS table) in the caller's own process, and a miss falls through to a single fill that checks L2 (Valkey), runs the declared loader, and writes both layers. The read path is the cache's whole job — one value, read often, held close.

# echo/apps/echo_cache — EchoCache.Table.fetch/3
# Read through the cache: an L1 hit never enters the owner process;
# a miss is a single-flight fill through the owner (L2, then the loader).
def fetch(name, id, timeout \\ 10_000) do
  case EchoCache.spec(name) do
    :error -> {:error, :no_such_cache}
    {:ok, spec} ->
      with :ok <- gate(spec.kind, id) do          # wrong-namespace id refused at the door
        now = System.monotonic_time(:millisecond)

        case :ets.lookup(name, id) do
          [{^id, value, expires_at, _v}] when now < expires_at ->
            {:ok, value, :hit}                      # L1 hit, in the caller's process
          _ ->
            GenServer.call(name, {:fill, id}, timeout)  # single-flight fill: :l2 or :fill
        end
      end
  end
end

Three messaging shapes

The three pillars are three classic messaging shapes, and naming the shape is the fastest way to know which pillar a need belongs to. A queue is point to point: a message goes to one consumer, and a second consumer competing on the same queue gets a different message, never a copy. A bus is one to many: a message goes to every subscriber, each getting its own copy. A cache is neither — it is a read path: the same value is served to whoever asks, again and again, until it expires.

The Queue
Point to point. One job is claimed by exactly one worker. Competing workers share the load; no worker sees another's job.
The Bus
One to many. One event is delivered to every subscriber, each with its own copy. A new subscriber changes nothing for the others.
The Cache
Read through. One value is served to every reader until it expires. A miss fills once; the herd reads the single answer.

Name the shape, and the pillar names itself. One job to one worker is the Queue; one fact to many listeners is the Bus; one value read often is the Cache.

the pattern (redis-patterns)

The Redis pattern catalog teaches each shape on its own — reliable queues for work distribution, streams and events for broadcast and replay, the cache-aside family for the read path. Three patterns, taught separately.

the pillars (EchoMQ)

EchoMQ applies all three over one owned keyspace and one Lua layer — EchoMQ.Jobs.enqueue/4 + claim/3, EchoMQ.Events.subscribe/2 + publish/5, and EchoCache.Table.fetch/3. Three surfaces, one substrate.

Which pillar fits the need

Match a real need to its pillar. Pick a need below; the readout names the pillar, the shape it fits, and the surface that does it. The rule is the shape: does the data go to one worker, to many listeners, or back to readers until it goes stale?

A need · which pillar fits it

Some needs ride more than one pillar — an order that settles is broadcast on the Bus and may invalidate a cached read. The Overview names the surfaces; the pillar chapters show how they compose. Next: the protocol below the line that lets them share one wire.

Recap — the three pillars

EchoMQ is one system with three surfaces over one owned wire. The Queue distributes work point to point (EchoMQ.Jobs.enqueue/4, claim/3, EchoMQ.Consumer). The Bus broadcasts signals one to many (EchoMQ.Events.subscribe/2, publish/5) and keeps a replayable log. The Cache serves reads through two layers (EchoCache.Table.fetch/3). Each is named here and taught in depth in its own pillar chapter. The next dive draws the line that lets all three share one wire — the protocol below the language line.

Next in the Overview: The protocol below the line — why the three pillars interoperate, and why any runtime can speak them.

References

Where the three pillars are grounded, and where the engine and the dispatch they share are documented.

Sources

  • Valkey — Documentation — the BSD-licensed, foundation-governed store EchoMQ is backed by — the substrate of record for all three pillars.
  • Redis — EVALSHA — the load-once, run-by-SHA dispatch each pillar's atomic script runs through.
  • Redis — Documentation — the data structures the keyspace under the three pillars is built from.
  • DragonflyDB — Server flags — --lock_on_hashtags: the thread-per-shard engine the declared-key, per-queue-hashtag keyspace is built for.

Related in this course