At-least-once

A reliable queue can hand the same job to a worker more than once. What separates a correct consumer from a broken one is whether the effect stays at one no matter how many times the job arrives. The toggle below replays one job up to three times against two consumers: an idempotent one that claims a dedup marker with SET … NX before the effect, and a non-idempotent one with no marker. The readout reports deliveries and effects side by side.

A separate monitor process — a reaper — periodically scans the processing list for stalled messages. For each job that has sat there longer than a timeout, it treats the worker as dead and moves the job back to the main queue with a second atomic move. R3.01 grounded this in EchoMQ's stalled sweep. The recovery path is what makes the queue reliable, and it is also what makes redelivery inevitable.

The window is precise. A worker takes a job into the in-flight list under a lock, applies the effect — charges a card, writes a row, sends a mail — and then crashes before it can remove the job from the in-flight list. The effect already happened; the acknowledgement did not. The reaper finds a job whose lock expired, returns it to the main queue, and a second worker runs it again. The effect runs twice.

This guarantees “at-least-once” delivery — messages may be processed multiple times but are never lost.

That sentence from the source is the whole bargain. The dependency chain is exact: R3.01's in-flight move makes redelivery possible (that is the design); a finish-then-crash before the acknowledgement makes it inevitable; therefore the consumer must be idempotent. There is no fourth option that removes the duplicate without reintroducing the loss.

This pattern provides at-least-once delivery, which is two properties stated together:

Messages are never lost. A job always exists in some list — the main queue or the in-flight list — so no crash drops it.

Messages may be processed multiple times. A worker that crashes after the effect but before the acknowledgement leaves the job in-flight, and recovery runs it again.

The three guarantees that bound the design space are worth naming together. At-most-once removes the job before the work, so a crash loses it — zero or one delivery, never a duplicate, but a job can vanish. At-least-once keeps the job until the acknowledgement, so a crash redelivers it — one or more deliveries, never zero, but a job can repeat. Exactly-once would be one delivery and exactly one — and it is not achievable as a delivery guarantee, because the acknowledgement itself can be lost, which is the subject of the third dive.

For exactly-once semantics, make message handlers idempotent — processing the same message twice should produce the same result as processing it once.

This is where the work moves to the consumer. Idempotency converts at-least-once delivery into an exactly-once effect. Some effects are naturally idempotent — a SET key value overwrites to the same value however many times it runs. Others are not — an INCR, a charge, a row insert — and those need a guard: a dedup marker claimed with SET … NX keyed by a stable job id, so the second delivery finds the marker and skips.

EchoMQ carries a real deduplication key, and it is worth being precise about what it does. EchoMQ.Keys.dedup/2 builds emq:{queue}:de:{id} ("#{base(ctx)}:de:#{dedup_id}"). A producer enqueuing a job with a deduplication id claims this marker; a second enqueue carrying the same id is dropped because the marker is already present. The Lua script removeDeduplicationKey-1.lua cleans it up: it GETs the key and DELs it only when the stored value matches the job id, so the cleanup never deletes a marker some other job holds.

# at-least-once delivery + an idempotent effect = exactly-once effect (the consumer's job)
if SET dedup:{job_id} 1 NX:     # claim the marker; NX = only if absent
    apply_effect(job)           # first delivery: run the effect once
else:
    skip()                      # a redelivery: the marker is already set, do nothing

-- EchoMQ.Keys.dedup/2 — producer-side dedup key (real)
def dedup(ctx, dedup_id), do: "#{base(ctx)}:de:#{dedup_id}"   # emq:my_queue:de:<id>

This is producer-side deduplication. It stops two identical jobs from entering the queue at enqueue time. It is not consumer exactly-once: it says nothing about a single job being delivered twice to a worker, because the in-flight recovery path can still redeliver a job that entered the queue once. Producer dedup and consumer idempotency are two different keys protecting two different moments — the enqueue and the effect. The exactly-once effect a reliable queue ships is the consumer's job: a naturally idempotent effect, or a marker claimed at effect-time with SET … NX.

Each dive takes one face of the bargain, in order: the three guarantees and why the move-under-lock yields at-least-once; making the consumer idempotent so a redelivery changes nothing; and the acknowledgement-gap argument that proves exactly-once delivery cannot exist. Read them in order — the semantics first, the idempotent consumer second, the impossibility proof third.

The Redis mechanisms behind at-least-once delivery and the idempotency marker, and the queue protocol whose producer-side dedup key grounds this module.