Jitter & the thundering herd

A dependency goes down. Every job that needed it fails in the same window — say a thousand jobs failing within the same second. Each runs the same exponential formula with the same base and the same attempt number, so each computes the same delay. They all re-schedule for the same future millisecond. When that millisecond arrives, all thousand re-fire together.

The dependency, perhaps newly back on its feet, takes the full thousand-job spike in one instant and may fail again — which schedules a thousand more retries for the next backoff window, and the spike repeats. Backoff was meant to space the load out; without jitter it re-synchronizes it. The herd is the cost of every member computing an identical delay from identical inputs.

The fix is to add controlled randomness to each delay so the retries land across a window instead of on a point. EchoMQ.Backoff applies it in apply_jitter/2. For a jitter fraction in (0, 1] it widens the delay into a band.

# EchoMQ.Backoff.apply_jitter/2 — the jitter clause (real)
defp apply_jitter(delay, jitter) when jitter > 0 and jitter <= 1 do
  min_delay    = trunc(delay * (1 - jitter))
  jitter_range = trunc(delay * jitter * 2)
  min_delay + :rand.uniform(jitter_range + 1) - 1
end

min_delay is the floor — the computed delay reduced by the jitter fraction. jitter_range spans twice the fraction, so the band runs from delay × (1 - jitter) up to delay × (1 + jitter). :rand.uniform picks a point inside it, independently for each job. With jitter: 0.2 on a 4000 ms delay, min_delay is 3200, jitter_range is 1600, and each retry lands somewhere in [3200, 4800] — the computed delay plus or minus twenty percent. A thousand jobs that would have re-fired on one millisecond now scatter across a 1600 ms window, a few per millisecond instead of all at once. The interactive below sweeps the jitter fraction and reads the band it produces.

The herd is not exclusive to exponential schedules. A fixed backoff — the same base every attempt — has the same flaw: a batch that fails together re-fires together. EchoMQ.Backoff's :fixed clause routes through the same jitter: apply_jitter(base_delay, jitter). So a %{type: :fixed, delay: 5_000, jitter: 0.2} config spaces a batch's retries across [4000, 6000] exactly the way the exponential one does. Jitter is the herd-breaker; the strategy (fixed or exponential) only sets the center of the band, not whether it is spread.

The spread is real code in echo/apps/echomq. Both built-in strategies end in apply_jitter(delay, jitter): the :exponential clause computes trunc(:math.pow(2, attempt - 1) × base_delay) first, the :fixed clause passes the base straight through. The jitter clause then returns min_delay + :rand.uniform(jitter_range + 1) - 1. The doc example carries it: calculate(:exponential, 3, 1000, jitter: 0.2) returns about 4000 — about, not exactly, because each call draws a different point in the band. The default config sets jitter: 0.2, so the spread is on by default. A jitter of 0 (the apply_jitter(delay, 0) clause) returns the delay unchanged — that is the synchronized case, the herd.

# jitter: 0.2 on a 4000ms delay — the band (real arithmetic)
min_delay    = trunc(4000 * (1 - 0.2))   # => 3200
jitter_range = trunc(4000 * 0.2 * 2)     # => 1600
delay        = 3200 + :rand.uniform(1601) - 1  # => a point in [3200, 4800]

The write that places each jittered retry at its own fire-time, the sorted set the spread scatters across, and the queue protocol EchoMQ ports.