## Summary <!-- REPLACE: your own words. Draft below for you to edit. --> Failure handling is the weakest part of vtasks and needs a design, not more patches. Every open issue (#6–#10) lives in this area, and the last one made it clear that the individual bugs are symptoms of two missing concepts rather than independent defects. **AI disclosure:** Claude Code — investigated the current state, verified each claim below against the source, and drafted this issue. ## The two things that don't exist **There is no retry.** `_build_task_data` writes `"retries": 0` into every payload (`backends/base.py`) and nothing ever reads or increments it. `max_retries` does not appear anywhere in the package. Any exception in a task goes straight to the DLQ via `_fail_task` — no attempt count, no backoff, no second chance. PROTOCOL.md is honest about it: `"retries": 0, // reserved; send 0`. **Nothing comes back out of the DLQ.** `clear_queue --failed` deletes; `admin_valkey.py` displays. There is no requeue path, and the DLQ is a capped list (`LPUSH` + `LTRIM` to `VTASKS_DLQ_CAP`), so entries are silently evicted oldest-first once it fills. Because retry doesn't exist, `max_rescues` has been pressed into service as the only failure budget users have. But rescue is *redelivery after worker death*, which has the opposite risk profile to *retry after an exception*: a task that OOM-killed its worker must not be blindly redelivered, while a task that hit a transient `ConnectionError` should be. One counter and one default cannot serve both, and that conflation is the root of #10 rather than the call-site bug that triggered it. ## Ownership is inferred, not recorded Rescue decides "is this task abandoned?" from a *process* name (`{hostname}-{pid}`) plus a heartbeat marker, instead of from the task. That indirection is where the remaining sharp edges come from: - The worker's own record means "orphaned" at startup and "executing" during the sweep, distinguishable only by call site — the cause of #10. - The DB backend cannot reclaim its own predecessor's rows after a restart (#9), because the only way it can avoid #10's bug is to exclude its own ID always. - Two live workers sharing an ID break the model entirely. - Both scans are hostname-scoped, so a Kubernetes pod replacement (new pod name on every rollout) strands `processing:{old-pod}-{pid}` keys that nothing will ever match — one leaked key per killed pod, tasks neither run nor recorded. Worth stating plainly, because it rules out a family of tempting shortcuts: an ID unique enough that two live workers cannot collide is also unique enough that a restarted worker cannot recognise its predecessor. Those are the same property. Naming cannot resolve this; only per-task ownership can. ## What a design should settle 1. **Separate the two budgets.** `retries`/`max_retries` for task-level exceptions, `_rescue_count`/`max_rescues` for post-crash redelivery. They must not share a counter. 2. **Per-task lease instead of process inference.** The task record carries its claimant and an expiry; recovery reclaims expired leases and needs no hostname or PID reasoning. This subsumes #9, #10, the shared-ID question and the cross-hostname leak, and demotes `worker_id` to an observability label. 3. **Retry policy.** Backoff shape, jitter, per-task overrides, and whether a retried task keeps its original ID. 4. **The semantic question, and it is the important one:** should retry preserve the no-double-execution stance that `max_rescues=0` currently defaults to? At-least-once redelivery and attempt-tracked-with-budget are different products. Answering this first determines everything above. 5. **DLQ as a first-class store.** A requeue path, and whether a capped list is right at all given it silently evicts real failures. ## Two things that make this cheaper than it looks - `retries` is already a **reserved** protocol field and PROTOCOL.md says "omitted fields generally default as shown", so adding retry semantics is a compatible extension — **not** a protocol v2 break. - Retry with backoff needs no new machinery: it is re-enqueue with `run_after = now + backoff`, and the delayed-task substrate already exists on both backends (per-queue scheduled sorted-set plus promoter on Valkey, indexed `run_after` column on the DB backend). DLQ requeue is also independently shippable ahead of any of this: DLQ entries retain the full `func`/`args`/`kwargs`, so a `requeue_failed` command needs no design decisions and no protocol change. ## Adjacent, cheap, and worth doing regardless `_ack_task` discards the `LREM` return value (`backends/valkey.py`), so a worker whose task was reclaimed out from under it acks into the void and reports success. This is *why #10 went unnoticed for so long* — the only symptom was odd DLQ entries. Checking the count and emitting a warning plus a counter metric would have surfaced it immediately, and would surface lease-stealing under any future design.