Why exactly-once is a lie

The argument is the two-generals problem in queue clothing. A worker takes a job, does the work, and sends an acknowledgement. The queue removes the job when the acknowledgement arrives. Now ask: what happens if the acknowledgement is lost in transit, or the worker crashes the instant after sending it.

The queue is in a position with no clean exit. It sent the job and received no acknowledgement. Two histories produce that exact state, and from the outside the two are indistinguishable:

The work never ran (a real loss), so the job must be redelivered. Or the work ran and the acknowledgement was lost, so redelivering runs the work twice.

To guarantee no loss, the queue must redeliver on a missing acknowledgement. But redelivering on a missing acknowledgement runs the second history twice. There is no third behaviour: redeliver and risk a duplicate, or do not redeliver and risk a loss. A confirmation of the confirmation does not help — that message can be lost too, and the regress never bottoms out. The gap is structural, not an implementation gap.

worker:  take job → do work → send ACK ──╳ (ACK lost here)
queue:   sent job, no ACK on record → must redeliver to avoid loss → work runs twice
         # "work never ran" and "ACK was lost" leave the same state

So a queue picks a side. A reliable queue picks no-loss, which forces redeliver-on-missing-ack, which is at-least-once. The duplicate is the price of never losing a job, and it is paid at the acknowledgement gap.

Exactly-once delivery is unreachable; exactly-once effect is reachable, and it is what the phrase should mean. Hold at-least-once delivery — never lose a job — and make the consumer idempotent so a redelivery changes nothing the second time. The effect happens once even though the delivery may happen twice. Systems that advertise “exactly-once” deliver this: at-least-once transport plus an idempotent or deduplicating consumer. The honest framing moves the guarantee from the delivery to the effect, where it can actually hold.

This reframing is not a downgrade. It is more honest and it is more robust, because it puts the duplicate-handling where the duplicate is observable — at the effect, keyed by the job's identity — rather than pretending a transport layer can resolve a lost acknowledgement.

The sharpest error in this area is to point at a deduplication key and call it exactly-once. EchoMQ's EchoMQ.Keys.dedup/2 builds emq:{queue}:de:{id} ("#{base(ctx)}:de:#{dedup_id}"), and removeDeduplicationKey-1.lua cleans it up — it GETs the key and DELs it only when the value matches the job id. This is real and useful, and it is producer-side: it dedups at enqueue, dropping a second job that carries the same deduplication id before it ever enters the queue.

That is a different job from consumer exactly-once. Producer dedup answers “two identical jobs were submitted — keep one.” Consumer idempotency answers “one job was delivered twice by the recovery path — apply its effect once.” The recovery sweep can redeliver a single in-flight job whose lock expired regardless of the producer dedup marker, because that job already entered the queue exactly once. So the de:{id} marker does nothing to close the acknowledgement gap at the consumer.

The grounding is precise, and it is the discipline of this whole module. EchoMQ's EchoMQ.Keys.dedup/2 is a real, producer-side key — it dedups submissions at enqueue and is cleaned by removeDeduplicationKey-1.lua. It is not, and does not claim to be, consumer exactly-once. The exactly-once effect a reliable queue ships is the consumer's, made by an effect-time NX marker or a naturally idempotent effect, as the previous dive built.

The command behind the consumer-side exactly-once effect, the queue's delivery-guarantee framing, the reliable-queue design note, and the protocol EchoMQ ports.