fix: don't let the periodic rescue sweep DLQ in-flight tasks

Summary

Fixes #10 (closed). Hotfix scope: the reported bug and nothing else.

_fetch_task BLMOVEs a task into processing:{worker_id} and only a terminal outcome removes it (_ack_task on success, _fail_task on failure), so while a task is executing it sits in that key. _rescue_tasks() drained that key unconditionally, and worker.py called the same function from two call sites that disagree about what the own key means:

call site	what the own key holds	correct action
startup scan	orphans from a previous incarnation at the same {hostname}-{pid}	drain it
periodic sweep	tasks executing right now	leave it alone

So on every sweep the lock holder popped its own in-flight tasks, evaluated _rescue_count (0) >= max_rescues (0) and dead-lettered them — while going on to finish them normally:

Processing task w5aYrB…: apps.uptime.tasks.dispatch_checks
Task w5aYrB… exceeded max rescues (0/0), sending to DLQ      ← sweep, mid-execution
Task w5aYrB… finished in 0.0061s                             ← same task, completes fine

This is one worker robbing itself — there is no second worker involved, so it is not a multi-worker or shared-ID question. The liveness check cannot help: it distinguishes siblings, and a worker is never its own dead sibling.

_rescue_tasks() now takes include_own and the sweep passes include_own=False. The startup scan is unchanged.

Why this was worse than the log noise suggests

The DLQ is a capped list (LPUSH + LTRIM to VTASKS_DLQ_CAP, default 1000), so every false entry evicts a genuine failure. The reporter's "not a single true positive amongst them" is the real severity: the DLQ stopped being a failure record.

Raising VTASKS_MAX_RESCUES was not a workaround. Confirmed against real Valkey that it moves the task into the re-enqueue branch instead, handing a still-executing task to another worker — genuine double execution.

The database backend never had this bug

It excludes its own rows unconditionally, so its sweep could not touch its own in-flight work. db.py is at its main behaviour here; it takes include_own only for parity with the backend contract.

Scope: what is deliberately not here

An earlier revision of this branch also tried to fix #9 (closed) (the DB backend not reclaiming its own predecessor's rows). That has been removed. It was the half carrying every judgment call — whether a worker may assume its own ID's records are dead, what happens when two live workers share an ID, whether the generated ID should carry a uniqueness nonce — and none of it belongs in a hotfix.

It is also the wrong layer. An ID unique enough that two live workers cannot collide is also unique enough that a restarted worker cannot recognise its predecessor; those are the same property, so no naming scheme resolves it. The fix is to stop inferring task ownership from a process name plus a heartbeat and record it on the task with a lease — see #11, which covers the failure/retry rework this belongs to. Comments and docs now point at the gap where it is visible rather than papering over it.

Also in here

• _rescue_tasks is now declared on VTasksBaseBackend. It was the one lifecycle hook missing from the contract, so changing its signature broke out-of-tree backends silently: worker.py swallows sweep failures as a warning, so an old-signature backend raises TypeError every 30s with rescue permanently dead and nothing saying why. tests/test_metrics_integration.py was an in-tree example of exactly that shape. Breaking for backend implementers — the keyword is required.
• include_own is a required keyword rather than defaulting to True. A default would make "drain my own record" what you get by forgetting the argument.
• The "Total rescued tasks on startup" log line fired on periodic sweeps too, and the startup-scan failure now logs at error saying what is left unclaimed — with the sweep off the own record, that scan is the only path to it.
• PROTOCOL.md §7 described the alive: marker as the whole guarantee, which is what made the old behaviour look correct. No wire-format change; protocol stays v1.
• Documentation accuracy, found while verifying the above: docs/deployment.md claimed twice that rescue "re-enqueues" crashed tasks, which is false at the default VTASKS_MAX_RESCUES=0 where it dead-letters them — so an operator tuning GRANIAN_WORKERS_KILL_TIMEOUT was being told SIGKILLed work is recovered when by default it is discarded. Also: VTASKS_COMPRESS_THRESHOLD was still in the settings reference though 3.0.0 removed it; the three settings that actually govern crash recovery were absent from it; and the worker-ID contract is now stated explicitly.

Testing

Five new tests at three levels. Each was checked to fail against the unfixed source before being trusted:

• tests/integration/test_valkey.py (real Valkey): the production scenario — enqueue, _fetch_task, sweep, assert the task is still in flight with an empty DLQ, then ack and confirm the ack still clears the key. Plus the counterpart that the startup scan still drains the own key, with a fresh own liveness marker present.
• tests/test_rescue.py (mocked driver): the sweep never pops the own key but still reclaims a dead sibling; the startup scan does pop it.
• tests/test_rescue.py (DB): own rows excluded on both scans, pinning current behaviour so a silent change to it becomes visible.
• WorkerBackgroundLoopTests: the two call sites pass different include_own values, and the sweep stays behind vtasks_rescue_lock. This is the regression that actually caused the bug, so it is the one guarding against a repeat — verified by regressing the call site and watching the suite fail.

test_does_not_rescue_own_tasks previously asserted "tasks belonging to the current worker should NOT be rescued" with no argument; it now states which scan it means.

Suite results: 184 OK on Postgres 18 + Valkey (includes the valkey-tagged integration tests, run against a live broker via docker compose), 165 OK (11 skipped) on SQLite. Separately re-ran the standalone reproduction that generated the reported Exceeded max_rescues (0) / _rescue_count: 1 entry — it now leaves the task in flight, and the max_rescues=5 duplicate-delivery variant no longer re-enqueues.

AI disclosure: Claude Code — reproduced the issue, wrote the fix and tests, ran the verification, ran an adversarial self-review and applied its findings, and drafted this description. Reviewed by me before pushing.