Service metrics can lead to performance issues and the worker cannot recover to a normal state. It becomes impossible to run new workflows on this worker. ### Steps to reproduce Create a fresh cluster, and run a worfklow that will induce a lot of load on the cluster. E.G: emit every 1s (up to 200 times) \>\> cat something \>\> sleep 5s. ### What is the current _bug_ behavior and the expected _correct_ behavior? Everything suggests that it comes from the metrics_service.py, and more particularly the function `periodic_fetch_metrics_for_all_executions` that acquires a lock and that can take seconds (up to 4s at the time I am writing this line) to execute. Since this function is called every seconds (https://gitlab.com/ryax-tech/ryax/ryax-runner/-/blob/master/ryax/worker/app.py?ref_type=heads#L271), the coroutines start to stack up in the event loop, and the calls will be handled one by one. On the other hand, this aforementioned lock is necessary to be able to administrate the cluster (i.e., starting/ending execution). But the calls (for instance, `ryax.worker.application.execution_handlers.on_create_execution`) are waiting for the same lock as the one used by `periodic_fetch_metrics_for_all_executions`. This can generate a waiting time of hours. The precise function taking too long, is the `StartMonitoringExecutionCommand` as depicted in the logs below. The two following log lines show that it takes 2 hours to complete the `StartMonitorintExecutionCommand` command. ```json { "line": "2026-03-20 18:49:35,240 DEBUG [ryax.common.application.message_bus] [message_bus.py:214] [trace_id=86e2b40fbd7e2dc7d1e21c78258d215e span_id=8d75c87fa6f1b70d resource.service.name= trace_sampled=True] - Start handling command StartMonitoringExecutionCommand(execution_id='ExecutionConnection-1774024213-1zvc4hec') with handler 'ryax.worker.application.metrics_handler.on_start_monitoring_execution_command'", "timestamp": "1774028975240807023", "date": "2026-03-20T17:49:35.240Z", "fields": { "app": "worker", "component": "internal", "container": "ryax-worker", "filename": "/var/log/pods/ryaxns_ryax-worker-68f7fd869c-z9h92_201e881b-c9cd-463c-bed1-7b6ff8bb6cf7/ryax-worker/0.log", "helm_sh_chart": "worker-26.2.1", "instance": "ryax", "job": "ryaxns/worker", "namespace": "ryaxns", "node_name": "scw-dev1-default-c00f04d1b2194d8c8a194d259e896", "pod": "ryax-worker-68f7fd869c-z9h92", "pod_template_hash": "68f7fd869c", "ryax_tech_resource_name": "worker", "ryax_tech_resource_type": "internal", "service_name": "worker", "stream": "stderr" } }, { "line": "2026-03-20 20:48:11,128 DEBUG [ryax.common.application.message_bus] [message_bus.py:249] [trace_id=39f0b9bba8c63375467eef8a9c9634a4 span_id=42e5a60b15d3d889 resource.service.name= trace_sampled=True] - End handling command StartMonitoringExecutionCommand(execution_id='ExecutionConnection-1774024213-1zvc4hec') with handler 'ryax.worker.application.metrics_handler.on_start_monitoring_execution_command'", "timestamp": "1774036091128701049", "date": "2026-03-20T19:48:11.128Z", "fields": { "app": "worker", "component": "internal", "container": "ryax-worker", "filename": "/var/log/pods/ryaxns_ryax-worker-68f7fd869c-z9h92_201e881b-c9cd-463c-bed1-7b6ff8bb6cf7/ryax-worker/0.log", "helm_sh_chart": "worker-26.2.1", "instance": "ryax", "job": "ryaxns/worker", "namespace": "ryaxns", "node_name": "scw-dev1-default-c00f04d1b2194d8c8a194d259e896", "pod": "ryax-worker-68f7fd869c-z9h92", "pod_template_hash": "68f7fd869c", "ryax_tech_resource_name": "worker", "ryax_tech_resource_type": "internal", "service_name": "worker", "stream": "stderr" } }, ``` ### Relevant logs and/or screenshots Graph of the ram usage on the worker: {width="900" height="352"} ### Link to the Ryax instance dev-1.ryax.io ### Possible fixes