Execution stay in Running but they are crashed, leaving deployment behind
Some executions crashed, but this is not captured by the worker. Thus, the action deployments stay allocated and is never released, keeping resources forever.
Steps to reproduce
Not sure... It seems that stressing the cluster with a lot of parallel actions triggers the issue.
What is the current bug behavior and the expected correct behavior?
The worker does not detect the execution failure and never trigger the timeout. The execution is still running even after a failure and after the timeout is reached. This should not happen ! The worker should detect the action failure, and even if the Worker fails to do this the actions should be undeployed at least after it reached the timeout.
Relevant logs and/or screenshots
On app.ryax.io some actions that are started for days, even month, with an allotted time of 900s, and they are still in Running state for both execution connection and it's allocated action deployment.
runner=> select action_deployment.id,execution_connection.id as exec_id ,action_deployment.state as action_state, execution_connection.state as exec_state,started_at,time_allotment as exec_state from action_deployment join execution_connection on allocated_connection_id = execution_conne
id | exec_id | action_state | exec_state | started_at | exec_state
--------------------------------------+-----------------------------------------+--------------+------------+-------------------------------+------------
ActionDeployment-1731430450-wcp6svdz | ExecutionConnection-1731430450-z7v2b0pa | RUNNING | RUNNING | 2024-11-12 16:54:13.524269+00 | 900
ActionDeployment-1729787136-cnzuxzc5 | ExecutionConnection-1729787136-m1q7y7vx | RUNNING | RUNNING | 2024-10-24 16:25:38.693136+00 | 900
ActionDeployment-1732547089-wpa3col1 | ExecutionConnection-1732547089-lgo0u5f2 | RUNNING | RUNNING | 2024-11-25 15:04:51.970275+00 | 900
ActionDeployment-1732803447-p3cwgzof | ExecutionConnection-1732803447-xs897kfc | RUNNING | RUNNING | 2024-11-28 14:17:30.047147+00 | 900Link to the Ryax instance
Possible fixes
Better handle the execution state to avoid missing failure events. also, create an independent process that garbage watch the execution in the database and enforce their timeout.