Executions stay RUNNING or STOPPING forever after their workflow is stopped
On core.ryax.io (26.9.0), 16 executions are still active on workflow deployments that were stopped. There are two causes, and both are still in the code.
1. Trigger fires stay RUNNING after their trigger ends (8 rows)
A trigger records each run it starts as a fire: an execution row with trigger_connection_id set. A synchronous trigger (the portal, the http_api addon) creates the fire as RUNNING when a request arrives, and the fire completes when its outputs come back.
When the trigger itself ends, on_ended (ryax/runner/application/handlers/execution_handlers.py) cancels the in-flight fire only in the CANCELED branch. A graceful stop (the trigger ends COMPLETED) and a crash (ERROR) leave the fire RUNNING forever, along with its workflow run. get_running_fire_of_trigger also raises not-found when more than one fire is in flight, so even a canceled trigger leaves concurrent requests behind.
| Fire | Trigger ended | Deployment |
|---|---|---|
ExecutionResult-1772619769-z3lo5n46 |
ERROR | K8s-CAPE-Prov-UI-Pulumi |
ExecutionResult-1772628231-lk2i1iq8, -1772628794-m8i21gej |
ERROR (same trigger) | K8s-CAPE-Prov-UI-Pulumi |
ExecutionResult-1772631613-vnesnk0i |
ERROR | K8s-CAPE-Prov-UI-Pulumi |
ExecutionResult-1779584815-n257pulb |
COMPLETED | video object detection |
ExecutionResult-1781794491-7ydwmu91 |
COMPLETED | CPU LLM Service |
ExecutionResult-1786356473-siog1duu |
ERROR | CPU LLM Service |
Execution-1790161942-qgsqtutt (26.9) |
COMPLETED | CAPE Qwen Next Pulumi Benchmark |
2. Stopped executions stay STOPPING (7 rows, 2026-09-14 to 09-23)
The Runner sends each stop to the right site and waits in STOPPING for the Worker's end event. Two Worker paths never send it.
The Worker does not know the execution (5 rows; they never started). on_stop_execution (ryax/worker_k8s/application/execution_handlers.py) raises an internal ExecutionEndedEvent(ERROR_INTERNAL, "Execution not found in the Worker"). But on_execution_ended_event first loads the execution and returns on ExecutionNotFoundError, so nothing reaches the Runner. The scaleway Worker's database has none of its 4 ids. The 5th is on nebius-k8s and was not checked.
The Worker restarted since the execution started (2 rows: Execution-1789566092-gqgorx3n and Execution-1789918237-uaq0l4w2). IExecutionService.stop cancels an in-memory asyncio task. After a restart there is none, so it logs "No task is registered … ignoring" and sends nothing. The Worker's database still records both as RUNNING, although their Kubernetes objects are gone. Both were started before a restart of ryax-core-worker (upgrade on 09-17, pod restart around 09-22) and stopped after it.
The execution reaper (bdd7ec4a) covers none of the 15 rows above. It skips executions that never started, root actions and fires.
3. Legacy (not fixed here)
ExecutionConnection-1779460813-i0ju65bi has been ALLOCATED since 2026-05-22 (26.4). cron-video-detection (WorkflowDeployment-1779719174-j5xx3yao) has been STARTED since 2026-05-25 while its trigger fired normally. The logs that would explain it are gone (Loki keeps 7 days). On 26.9 a normal undeploy clears it.
Found while investigating #1337 (closed).