Two-phase mark/recover

The first sweep that finds a job stalled does not reclaim it. It marks it: it records the job in a stalled set and moves on. The mark is a memory across sweeps — a note that this job was found without a live worker once. A job that was merely paused, whose worker renews its lock before the next sweep, has its mark cleared and is never touched. A job whose worker is genuinely dead stays stalled, and its mark survives.

The grace pass costs one sweep interval of delay before a dead worker's job is reclaimed, in exchange for not redelivering every job that briefly pauses. The trade is deliberate: a momentary stall is common and harmless, while a double-run is expensive, so the sweep spends one interval to avoid the second.

# sweep 1: mark the stalled job (no reclaim yet)
SADD emq:{queue}:stalled <id>             # record it; a paused worker clears this before sweep 2

The next sweep checks the marked jobs. A job still in the stalled set, still without a live worker, is recovered: its attempt count is incremented, and it is moved out of active. A job whose mark was cleared in between — its worker came back and renewed the lock — is left alone.

Where a recovered job goes depends on its attempt count. A job back from a stall has used one of its attempts. If it has attempts left, it returns to wait to be picked up again. If the recovery has pushed it past its maximum attempts, it goes to failed instead — a job that stalls every time it runs would otherwise loop forever, so the attempt ceiling caps the retries. Slide the attempts used below up to the ceiling to watch the recovered job switch from wait to failed.

# sweep 2: recover only if still marked and still stalled
if SISMEMBER emq:{queue}:stalled <id> and EXISTS emq:{queue}:<id>:lock == 0:
    attempts = HINCRBY emq:{queue}:<id> atm 1
    LREM emq:{queue}:active 1 <id>
    if attempts >= max_attempts:
        ZADD emq:{queue}:failed <ts> <id>   # past the ceiling: fail, do not retry
    else:
        LPUSH emq:{queue}:wait <id>          # attempts left: back to wait for another worker

In EchoMQ the two phases live inside one script. moveStalledJobsToWait-8.lua operates over a stalled set (EchoMQ.Keys.stalled/1 → emq:{queue}:stalled) alongside wait, active, and failed. On a sweep it marks a newly stalled job into the stalled set; a job already in the set that is still stalled is the one it recovers, moving it to wait or — past its attempt ceiling — to failed, incrementing the attempt count as it goes.

-- moveStalledJobsToWait-8.lua — the mark + recover, in one script (real, Elixir)
-- KEYS: stalled(SET) wait(LIST) active(LIST) failed(ZSET) stalled-check meta paused marker
-- ARGV[2] = max_stalled_count — how many stalls a job may survive before it is failed

The max_stalled_count argument to EchoMQ.Scripts.move_stalled_jobs_to_wait/4 bounds how many times a job may be recovered from a stall before it is failed outright. Because the mark and the recover live in the same script, the two-phase decision runs without a separate process holding intermediate state.

The set that marks a job once, the attempt-count increment, the scripting that keeps mark and recover in one step, and the queue protocol whose two-phase path EchoMQ ports.