Backoff & retry

When a job fails, the worker does not drop it and does not loop on it. It re-schedules the job to run again later — and "later" is a fire-time, the exact axis the delayed queue already orders by. The source shows the minimal form: re-add the failed job with a future score.

The member is the same job; the score is now + 60s. The delayed set sorts it by that fire-time alongside every other deferred job, and the due-sweep brings it back when the clock reaches it. The retry rides the machinery R4.01 already built — the only new question is what delay goes in the score. The interactive below fails a job at t = 0 and plots five exponential retries: move the base slider and read the next fire-time and the cumulative wall-clock.

A fixed delay treats a transient blip and a struggling dependency the same. A backoff delay grows with the attempt, so the first retry is quick and a repeatedly-failing dependency gets progressively more room. The exponential form doubles the delay each attempt.

# delay = base × 2^(attempt - 1)
attempt 1 → base × 2^0 = base
attempt 2 → base × 2^1 = 2 × base
attempt 3 → base × 2^2 = 4 × base

With a one-second base the retries land at 1s, 2s, 4s, 8s, 16s out. In EchoMQ this math is owned by EchoMQ.Backoff.calculate/4 — Elixir, not Lua. The :exponential clause computes delay = trunc(:math.pow(2, attempt - 1) × base_delay) then apply_jitter(delay, jitter). The doc example is exact: calculate(:exponential, 3, 1000, jitter: 0.2) returns about 4000. The :fixed strategy is the contrast — it returns apply_jitter(base_delay, jitter), the same base every attempt.

The ownership matters: the formula computes the delay value, and the reschedule script is handed that value. The Lua never computes 2^(n-1). The first dive walks the formula in full.

Exponential backoff alone has a failure mode. If many jobs fail at the same instant — a dependency that went down took them all out together — they all compute the same delay and all re-fire at the same future millisecond. The recovery becomes a synchronized spike: the thundering herd. The dependency, newly back on its feet, is hit by the whole batch at once.

Jitter spreads the retries. EchoMQ.Backoff applies it in apply_jitter/2: for a jitter fraction in (0, 1], it computes min_delay = trunc(delay × (1 - jitter)) and jitter_range = trunc(delay × jitter × 2), then returns min_delay + :rand.uniform(jitter_range + 1) - 1. With jitter: 0.2 on a 4000 ms delay, each retry lands somewhere in [3200, 4800] — the delay plus or minus twenty percent. The default config carries it: %{type: :exponential, delay: 1_000, jitter: 0.2}. A herd that would have re-fired in one spike is smeared across a window instead. The second dive runs the spread.

A retry is not a new mechanism. It is R4.01's machinery used a second time. EchoMQ.Scripts.retry_job/5 runs retryJob-11.lua, which re-adds the failed job to the delayed key (KEYS[7]) scored by the backoff fire-time. That script includes promoteDelayedJobs — the same due-sweep R4.01 teaches.

-- promoteDelayedJobs (included by retryJob-11.lua) — real
local jobs = rcall("ZRANGEBYSCORE", delayedKey, 0, (timestamp + 1) * 0x1000 - 1, "LIMIT", 0, 1000)
if (#jobs > 0) then
    rcall("ZREM", delayedKey, unpack(jobs))   -- then ZADD onto the wait / prioritized list
end

The retry re-enters emq:{queue}:delayed exactly the way a freshly-scheduled delayed job does — scored by getDelayedScore ((fire-time) × 0x1000) — and is swept back when the clock reaches it. R4.01 schedules; R4.04 sizes the delay with EchoMQ.Backoff and re-schedules on failure. The structure does not change between them. The third dive walks the reuse end to end.

A retry loop needs a floor. A job that fails forever should not retry forever. The source's own note is "maximum retry limits": track the attempt count in the job data and stop re-scheduling once it crosses a configured maximum. In EchoMQ the attempts limit is part of the job's options (attempts: 5 in the Backoff doc examples), checked on each failure before retry_job/5 is called; past the limit the job moves to the failed set instead of back to delayed. The backoff formula sizes the delays; the attempts cap fixes how many of them there are.

Backoff trades latency for protection. That trade is right for work that tolerates a delayed retry — an email, a webhook, a batch import — and wrong for work that must complete now or not at all. A user-facing request waiting on a synchronous response cannot afford an 8-second backoff before its second attempt; it needs a fast in-line retry or an immediate failure, not a deferred one. Backoff also assumes the failure is transient and the dependency will recover; against a permanent failure it only delays the inevitable trip to the failed set, so the attempts cap should be small.

For protecting a shared dependency from sustained load — not merely spacing one job's retries — rate limiting and the circuit breaker (R6 · Flow Control) are the right tools; backoff spaces one job's own retries, it does not cap the fleet.

Each dive takes one layer of retry-with-backoff, building on the source's retry idea and adding the EchoMQ grounding on top: the formula that sizes the delay, the jitter that spreads a batch, and the reuse of R4.01's delayed set. Read them in order — the formula first, the jitter second, the reuse third.

The Redis commands behind a retry — the re-add at a backoff fire-time, the due-sweep that brings it back, the sorted set both reuse — the queue protocol EchoMQ ports, the direction EchoMQ 2.0 takes native, and where the lifecycle that runs the retry is taught.