Kubernetes worker lost the metrics backpressure fix of #1396
The new Kubernetes worker (ryax/worker_k8s) lost the #1396 (closed) fix for metrics backpressure. It starts a new metrics fetch every second without waiting for the previous one. When a scrape takes longer than a second, the fetches pile up and slow the worker down. This is a regression from 26.9 for 26.10.0.
Steps to reproduce
- Run several executions at once on a K8s site, so that a metrics scrape takes more than 1 s.
- Watch the worker: fetch tasks accumulate and executions start or end more slowly.
What is the current bug behavior and the expected correct behavior?
- Current:
ryax/worker_k8s/app.pysleeps 1 s and then firesPeriodicFetchMetricsCommandwithblocking=False, so the fetches overlap without limit. - Expected: as in the old worker since #1396 (closed), there is at most one fetch at a time, aligned on the second, and a fetch that overruns its second is cancelled with a warning.
Relevant logs and/or screenshots
The old worker's loop, aec3eb77^:ryax/worker/app.py (lines 305-318), used asyncio.wait_for(handle_command(..., blocking=True), timeout=next - now). It was not carried over when the V1 worker was removed in aec3eb77.
Link to the Ryax instance
Found by reading the code, not on an instance.
Possible fixes
Port the old loop to worker_k8s and add a test.