A reliable enrollment queue

Portal enrolls a learner in a course. Frame that enrollment as a job: an enrollment request is enqueued, a worker processes it, and the job is completed once. “Reliable” means two things at the same time. A worker crash never drops an enrollment — the learner is enrolled, eventually, even if the first worker dies mid-process. And a worker crash never enrolls a learner into a corrupt half-state — no torn write, and no double-enrollment when the job runs twice.

Be plain about the boundary. Portal’s enroll is synchronous today and does not run through EchoMQ. EchoMQ is the candidate for Portal’s reserved F7–F9 multi-runtime layer. This workshop is the applied design: the reliable-queue path R3 built, laid over an enrollment job, so each pattern’s guarantee maps to a real failure it prevents. The Redis keys and Lua scripts are EchoMQ’s real ones; the wiring of enroll-as-a-queue-job is the design this page assembles, not Portal’s current architecture.

Five patterns assemble into one reliable loop. Each prevents a different failure of the naive form — pop a job, process it, done — and they build in order.

The processing list is the foundation: the enroll job moves from wait to active in one atomic step, so it is in exactly one list at every instant and a crash leaves it recoverable. That in-flight move is what makes redelivery possible — which is the point — and it is also what makes delivery at-least-once: a worker that enrolls the learner, then dies before it acknowledges, leaves the job in active to be redelivered. The stalled sweep reclaims that job; the atomic finish keeps the completion indivisible; the blocking pickup keeps the worker idle-cheap between jobs. Assembled, they are a queue that loses no job.

The interactive in the hero steps a single enroll job through the assembled loop over a fixed start state. Each press of step the job advances one stage of the reliable loop — wake, pick up under the lock, enroll, finish — and the readout shows the job’s list location and the enrolled count at each step. The function behind it is pure: it returns a new state and stores nothing.

Press crash mid-process at the enroll stage to inject the failure the chapter was built to survive. The worker dies after recording the enrollment but before the atomic finish. The job is stranded in active; its lock expires; the stalled sweep returns it to wait; a second worker picks it up and runs the enroll again. Because the enroll is idempotent, the second run returns {:error, :already_enrolled} and records nothing — and the readout shows enrolled: 1 even though the job ran twice.

Two behaviours are worth watching directly. The crash loses nothing — the job that died in active comes back through the sweep and finishes on the second worker. The redelivery corrupts nothing — the re-run hits the idempotency guard, so the enrolled count is one, not two. Both are the reliability and the idempotency the five R3 patterns were building toward, in one loop.

The assembled queue guarantees at-least-once delivery, so the enroll job can run twice. That is safe, because Portal’s enroll is already idempotent. Portal.Engine.Core.authorize/2 checks the folded enrollment state before any event is recorded:

A first enroll appends %Portal.Enrollment.Events.LearnerEnrolled{user_id, course_id, at} and the learner is enrolled. A redelivered enroll of the same learner into the same course hits the guard, returns {:error, :already_enrolled} — a real, closed Portal.Error code — and records nothing. Enrolling twice equals enrolled once.

The interactive below makes the proof concrete. Run the same enroll job any number of times; the enrolled count stays at one. The first run records the event; every later run returns {:error, :already_enrolled} before any write. At-least-once delivery over an idempotent effect is exactly-once-in-effect.

Every Redis primitive in the assembled path is real EchoMQ code in the iconic Elixir app echo/apps/echomq — the same symbols R3.01–R3.05 each cite. EchoMQ.Scripts.move_to_active/4 runs moveToActive-11.lua; EchoMQ.Scripts.move_stalled_jobs_to_wait/4 runs moveStalledJobsToWait-8.lua; EchoMQ.Scripts.move_to_finished/7 runs moveToFinished-14.lua (fourteen keys, fetch_next default one). The worker blocks on EchoMQ.Keys.marker/1 — emq:{queue}:marker, a ZSET — via BZPOPMIN (EchoMQ.Worker.wait_for_job/2), and a job add wakes it with ZADD marker 0 "0".

EchoMQ’s governing rule is “the Lua scripts ARE the protocol”: the same scripts are meant to run identically across the Elixir, Go, and Node.js runtimes. The honest status is that the three are not at parity — the iconic Elixir runtime is the most complete — so a guarantee holds wherever the script runs as one atomic step, strongest in Elixir.

The consumer is real Portal code in echo/apps/portal. The command is Portal.enroll/2 — the Portal facade function that delegates to Portal.Enrollment.enroll/2. The recorded fact is %Portal.Enrollment.Events.LearnerEnrolled{user_id, course_id, at}, emitted by Portal.Engine.Core.decide/2 and folded by evolve/2. The idempotency guard is Portal.Engine.Core.authorize/2, which returns {:error, :already_enrolled} against the folded state — the closed Portal.Error code that makes a redelivered enroll a safe no-op.

The Redis primitives the reliable enrollment queue assembles — the atomic in-flight move, the cached finish script, the blocking pickup, and the lock lease — and the queue protocol whose worker path EchoMQ ports.