## Summary A worker writes tasks which complete successfully into the DLQ. Locally observed: hundreds of such entries in 14 days, not a single true positive amongst them. I suspect the culprit to be the periodic rescue sweep in the Valkey backend running on defaults in >=3.x. ## Steps to Reproduce 1. Run a worker on the valkey backend with defaults (`VTASKS_RESCUE_INTERVAL=30`, `VTASKS_MAX_RESCUES=0`). 2. Keep a periodic task running frequently enough that one is in flight often — ours fires every second. 3. Watch `{vt}:q:failed` grow, and compare its entries against the worker log. ## Expected vs. Actual **Expected:** the DLQ contains tasks that actually failed. **Actual:** it fills with tasks that completed successfully. Over 14 days we collected 318 entries, every one carrying `error_msg: "Exceeded max_rescues (0)"`, `_rescue_count: 1`, `retries: 0`. Not a single genuine failure among them. --- **AI disclosure:** Claude Code running Opus 5. The analysis below was drafted with AI from a production instance and verified against the source; the summary above is mine. Also, I have reviewed the AI assisted part. ## Analysis (AI-assisted) `_rescue_tasks()` drains the worker's **own** processing key unconditionally (`backends/valkey.py:372-373`): ```python # Rescue from own key first (handles normal restarts) rescued = await self._rescue_processing_key(self._get_key(f"processing:{self.worker_id}")) ``` The liveness check that follows guards only *siblings* (`if sibling_id in alive: continue`). That is correct for the startup scan, where the own key genuinely holds orphans from a previous incarnation — and the closing log line still says so: `"Total rescued tasks on startup: %d"`. But `worker.py` calls the same function from two places: - `worker.py:320` — startup scan (own key is orphaned: correct) - `worker.py:437` — periodic sweep via `_rescue_loop` (own key holds **currently executing** tasks: incorrect) `_fetch_task` moves a task into `processing:{worker_id}` via `BLMOVE` and `_ack_task` only removes it on completion, so between fetch and ack every in-flight task sits in that key. The periodic sweep pops it, evaluates `rescue_count (0) >= max_rescues (0)` → true, and sends it to the DLQ while the worker finishes it normally. This appears to be a regression introduced with the periodic sweep in 3.0.0, whose changelog entry states the opposite intent: *"rescue only reclaims a sibling's in-flight tasks once that marker has expired, so a **live** worker's tasks are never stolen."* That guarantee holds for siblings but not for the worker itself. ### Evidence Worker log — sent to the DLQ, then completes: ``` Processing task w5aYrBFUuKeZRpo0itH8wWzM8KeOcakF: apps.uptime.tasks.dispatch_checks Task w5aYrBFUuKeZRpo0itH8wWzM8KeOcakF exceeded max rescues (0/0), sending to DLQ Task w5aYrBFUuKeZRpo0itH8wWzM8KeOcakF finished in 0.0061s ``` Timing fingerprint: our scheduler enqueues this task **every second**, yet the intervals between DLQ entries cluster on a 30-second grid — of 314 non-zero intervals, 236 have remainder 0 mod 30 and another 72 have remainder 1 (the drift of `asyncio.sleep`). That is 98% on the `VTASKS_RESCUE_INTERVAL` grid; the remaining six sit on remainders 2 and 29, i.e. also within a second or two of it. Genuine failures drawn from a 1-second stream would spread evenly across all 30 remainders. (`ts` is the enqueue timestamp, which for a task caught in flight is within milliseconds of the sweep.) The affected tasks track execution frequency rather than task identity: 312 × the per-second task, 3 × `perform_maintenance`, 3 × `process_event_alerts`. Setting `VTASKS_RESCUE_INTERVAL=0` stopped it immediately: the DLQ has not grown since, while the worker kept processing normally. ### The DB backend does the opposite `backends/db.py` excludes the own worker unconditionally: ```python .exclude(worker_id=self.worker_id) .exclude(worker_id__in=alive_ids) ``` So it never produces these false entries — but it also never rescues its own orphans after a restart, which looks like scenario 1 in #9 (stable hostname, worker reboots, marks itself alive, then cannot find its own orphaned task). Neither extreme is right: the own key *is* orphaned at startup and *is not* during the periodic sweep. ### Suggested fix Make the distinction explicit instead of relying on the call site: ```python async def _rescue_tasks(self, include_own: bool = True) -> None: ... if include_own: rescued = await self._rescue_processing_key( self._get_key(f"processing:{self.worker_id}") ) ``` and in `_rescue_loop` (`worker.py:437`): ```python await self.backend._rescue_tasks(include_own=False) ``` For `backends/db.py` the mirror image: apply `.exclude(worker_id=self.worker_id)` only when `include_own` is false. That keeps the periodic sweep safe and lets the startup scan recover the worker's own orphans — which is what #9 scenario 1 needs. Note that raising `VTASKS_MAX_RESCUES` is not a workaround: it moves the task into the re-enqueue branch (`valkey.py:441-444`) and causes genuine double execution. ### Environment `django-vtasks==3.1.0` (current `main` is byte-identical in `backends/valkey.py`), valkey backend, single standalone worker via `manage.py runworker --scheduler`, GlitchTip 6.2.6, Python 3.14, Valkey/Redis 7.4.7.