### Summary A single `ConnectionError` from the broker ends `Scheduler.run()` for the lifetime of the process. The worker in the same process recovers from the identical error and keeps going, so the pod stays healthy by every external signal and nothing is logged. Every scheduled task silently stops firing. For us that meant a self-hosted GlitchTip evaluated **no issue alerts at all for 3.5 days**. Errors were still captured and visible in the UI, so the only symptom was the absence of notifications, which is exactly the symptom nobody notices. ### Environment - `django-vtasks` 2.1.0, `django-vcache` 2.1.0, Python 3.14.3 - GlitchTip 6.1.4 (`glitchtip/glitchtip:6.1.4`), separate worker deployment (`embedWorker: false`) - Valkey broker reachable over the network (Kubernetes Service), single instance - Verified against `main` at `af041f3`: both code paths below are unchanged. ### What we observed | time (UTC) | event | |---|---| | 2026-07-27 19:27:21 | `Scheduler started.` Normal operation, `Enqueuing due task` every second. | | 2026-07-28 01:32:11.94 | Last `Enqueuing due task` ever emitted by this process. | | 2026-07-28 01:32:12.11 | `ConnectionError: broken pipe`, then `Connection refused (os error 111)`. | | 2026-07-28 02:11:53 | Broker pod recreated (node reboot). | | 2026-07-28 02:12:16 | The worker's queue consumers reconnect and resume normally. | | 2026-07-31 | Discovered by hand. The process had been up 4 days with 0 restarts. | The scheduler never resumed. Notably there is **no `Scheduler stopped.` line**, which a clean loop exit would have logged, so `run()` left through an exception rather than through `self.running = False`. ### Why it dies In `django_vtasks/scheduler.py`, the two backend calls at the top of the loop sit outside any `try`: ```python while self.running: logger.debug("Acquiring scheduler lock...") if not await self.backend.acquire_lock("vtasks_scheduler_lock", ttl=15): # <-- unguarded ... for task_name, task_config in self.schedule.items(): last_run_str = await self.backend.get_metadata(last_run_key) # <-- unguarded ``` The only `try` in the loop wraps the import-and-enqueue block further down. So a broker error raised by `acquire_lock` propagates out of `run()` and the coroutine is finished for good. Compare `Worker.consume_queue` in `django_vtasks/worker.py`, which wraps its whole loop body: ```python except asyncio.CancelledError: break except Exception: logger.error("Consumer error for queue %s", queue, exc_info=True) await asyncio.sleep(1) ``` Same process, same broker, same driver, same error, opposite outcomes. That asymmetry is the bug. Our logs show it directly: the `default` consumer logged `Consumer error for queue default`, backed off, and recovered, while the scheduler produced nothing and never came back. ### Why it was silent (the more important half) `django_vtasks/management/commands/runworker.py`: ```python worker_task = asyncio.create_task(worker.run(handle_signals=False)) scheduler_task = None if scheduler: scheduler_task = asyncio.create_task(scheduler.run()) await stop_event.wait() ``` Both Tasks are held in locals for the whole process lifetime and are only awaited during shutdown. asyncio's "Task exception was never retrieved" warning fires on garbage collection, and these Tasks are strongly referenced until the process exits, so it never fires. The traceback exists in the Task and is simply never looked at. The consequence is general: **any** unhandled exception in `worker.run()` or `Scheduler.run()` turns into a permanently half-dead process with zero log output. The scheduler is just the half that has no error handling of its own, so it is the half that hits this first. ### Why nothing external caught it - Pod `Running` / `Ready`, 0 restarts, for the whole 3.5 days. - `/tmp/worker_health` keeps being written, because `_heartbeat_loop` is a separate asyncio task that does not check whether the scheduler or any consumer is alive. An exec liveness probe on that file passes forever. - The scheduler emits no metric of its own, so there is nothing to alert on. We ended up detecting it externally, by reading `vtasks_last_run:<task>` out of Valkey and alerting on its age, plus auto-restarting the deployment when it goes stale. ### Suggested fix 1. **Guard the loop body in `Scheduler.run()`**, the way `consume_queue` already does: catch `Exception` per iteration, log with `exc_info=True`, sleep briefly, continue. A broker blip should cost a few scheduling ticks, not the scheduler. 2. **Observe the tasks in `runworker.main()`.** Either `add_done_callback` that logs the exception and sets `stop_event` so the process exits and the orchestrator restarts it, or race the tasks against the stop event with `asyncio.wait(..., return_when=FIRST_COMPLETED)`. Point 2 matters more than point 1: on its own it converts every future variant of this into a loud crash instead of a silent half-outage. 3. Optional but valuable: have the heartbeat file reflect the liveness of the subsystems rather than of the heartbeat loop, so a liveness probe can catch a half-dead process without an external check. Writing the file only while the scheduler and every consumer have ticked recently would be enough. We would rather see 2 than any amount of external monitoring on our side. Happy to open an MR for 1 and 2 if that is welcome. ### Reproducing 1. Run `runworker --scheduler` against a Valkey broker with at least one entry in `VTASKS_SCHEDULE`. 2. Once `Enqueuing due task` is flowing, stop the broker (`docker stop`, or drop the packets). 3. Bring it back after 30 seconds. 4. Observe: the queue consumers log a consumer error and recover; the scheduler logs nothing further, never emits `Scheduler stopped.`, and no scheduled task fires again for the life of the process. The process stays healthy and the heartbeat file keeps updating. ### Related The same "silent, permanent, half-dead process" shape bit us a second time three days later, in `fetch_batch` on the batch queue path, reported separately. Two instances in three days from different code paths is what makes point 2 above feel like the real fix.