### Summary After a Valkey outage, a `SERVER_ROLE=worker` process can run indefinitely with its **batch** queue consumer permanently not fetching, while the scheduler and non-batch queue consumers in the same process keep working normally. Nothing is logged. The process stays healthy by every external signal, including the `/tmp/worker_health` heartbeat, so orchestrators do not restart it. In our case this meant GlitchTip accepted events with HTTP 200 and showed none of them for 2h20m, because `ingest` is a batch queue. The events were not lost: they accumulated in the queue and drained as soon as the worker was restarted. The damage is the blind window, plus the risk that a broker restart during it discards the backlog. ### Environment - GlitchTip `6.1.4` (Docker image `glitchtip/glitchtip:6.1.4`), Python 3.14 - `django_vtasks` as shipped in that image - Valkey/Redis broker reachable over the network (Kubernetes Service), single instance - Separate web and worker deployments (`embedWorker: false`), so the worker is its own process ### What we observed The broker went away for about 34 minutes. Kubernetes evicted and recreated the worker pod during that window (node reboot). Three successive worker pods, from the same ReplicaSet: | worker pod | log window | log lines | ingest tasks processed | |---|---|---|---| | A | 00:24:00-00:25:18 | 284 | 4 (last at 00:25:03) | | B | 00:25:23-00:39:59 | 876 | **0** | | C | 01:00:10-02:22:34 | 4998 | **0** | | D (after manual restart) | 02:46 onward | healthy | working immediately | The broker was healthy again from **00:59:44**. Pod **C started at 01:00:10, 26 seconds after the broker recovered**, ran for 82 minutes, logged 4998 lines of ordinary scheduler and task activity, and processed zero batch-queue tasks. The queue was not empty: an external prober was enqueuing one event per minute, and we independently measured `LLEN {vt}:q:ingest` climbing steadily the entire time. Only a deliberate process restart fixed it. So this is not simply "a connection dropped and could not re-establish". In pods B and C the batch consumer appears never to have functioned at all, including against a fully healthy broker. ### The silence, and the asymmetry that we think explains it `ingest` was the only queue with a batch config, so it was the only one routed through `ValkeyTaskBackend.fetch_batch`. That method ends in: ```python # django_vtasks/backends/valkey.py try: result = await self.driver.blmpop(current_timeout, [key_name], "RIGHT", needed) ... except Exception: # Handle connection errors gracefully (e.g. timeout or pool closed) break ``` The bare `except ...: break` returns an empty list, which is indistinguishable from "no work available". So the error never reaches the handler in `Worker.consume_queue`: ```python # django_vtasks/worker.py except Exception: logger.error("Consumer error for queue %s", queue, exc_info=True) await asyncio.sleep(1) ``` The non-batch queues go through `_fetch_task`, where the exception *does* propagate to that handler. And that is exactly what we see in the logs: the `default` queue logged one ``` Consumer error for queue default ``` at 00:25:17 and then recovered and kept processing tasks for the rest of the outage. Same process, same broker, same driver, two code paths, opposite outcomes. Meanwhile the batch queue produced **no log line at all** across 2h20m of total failure. For reference, the underlying error at the time (from the web process, which surfaces it) was: ``` ConnectionError: Connection refused (os error 111) File "django_vcache/backend.py", line 473, in aget data = await self._driver.get(_key) ``` ### Why nothing caught it - Pod stays `Running` / `Ready`, 0 restarts. - `/tmp/worker_health` keeps being updated, because `_heartbeat_loop` is its own asyncio task and does not check whether any consumer is alive. So an exec liveness probe on that file passes forever. - `logger.debug` in `fetch_batch` only fires when tasks *were* collected, so raising the log level does not help. - The `QUEUE_DEPTH` gauge would have shown it, but it is only updated from inside `consume_queue`, and self-reported metrics from a broken consumer are not a great detector. We ended up detecting it externally, by watching the queue depth fail to decrease, and auto-restarting the worker deployment. ### Suggested fix The minimum useful change is to stop `fetch_batch` from hiding the error: 1. Let the exception propagate to `consume_queue`, which already logs it and backs off. If a swallow is wanted for genuine timeouts, catch the timeout type specifically rather than bare `Exception`, and `logger.warning` anything else before breaking. 2. Distinguish "timed out with no work" from "the fetch failed" in the return value, so the consumer can tell an idle queue from a broken one. Separately, and probably the more important half: something should make a permanently non-fetching consumer visible or fatal. A consumer that has failed N consecutive fetches could log at `error`, mark itself unhealthy in the heartbeat file, or exit the process so the orchestrator restarts it. Right now a worker with a dead batch consumer is indistinguishable from a healthy idle one. ### Reproducing We have not reduced this to a minimal script, and the exact trigger is the part we cannot explain: pod C started *after* the broker was healthy, so a simple startup race does not account for it. A plausible harness: 1. Run a worker with one batch queue and one non-batch queue. 2. Kill the broker. 3. While it is down, restart the worker process (or let an orchestrator recreate it). 4. Bring the broker back. 5. Enqueue to both queues. Expected: both drain. Observed for us: the non-batch queue drains, the batch queue never does, and nothing is logged. Happy to run diagnostics against a live reproduction if that would help, since we can trigger the conditions on our cluster.