Track bulk Duo workflow execution across asynchronous workers
What does this MR do and why?
This MR changes bulk Duo workflow execution to enqueue workflow workers asynchronously (perform_async) and introduces a shared WorkflowTrackable concern used by all vulnerability workflow workers.
The goal is to decouple workflow execution from the executor thread while ensuring every finding reaches a terminal state (completed, failed, or cancelled). Each worker is responsible for updating the execution state, scheduling the next batch when appropriate, and respecting execution cancellation before starting work.
Additionally, this MR:
- executes bulk Duo workflows asynchronously;
- introduces a shared worker concern for execution state tracking;
- propagates the execution ID across workflow workers;
- tracks completion and schedules subsequent batches from workers;
- handles worker failures and retry exhaustion by marking findings and executions as failed;
- prevents cancelled executions from starting already-enqueued work; and
- centralizes workflow-to-worker mappings used by bulk workflow execution.
Execution flow
StartService
│
├─ start execution
├─ schedule first batch
└─ enqueue workflow workers
└─ perform_async(item_id, execution_id)
│
▼
Workflow worker
│
├─ WorkflowTrackable
│
├─ fetch execution
│
├─ execution cancelled?
│ ├─ yes → mark finding cancelled
│ └─ no → execute workflow
│
├─ success
│ ├─ mark finding completed
│ └─ enqueue next batch (if any)
│
├─ retries exhausted
│ └─ mark finding failed and fail execution
│
└─ unrecoverable error (retries disabled)
└─ mark finding failed and fail executionThis allows bulk workflow execution to progress independently of the initial executor while ensuring execution state remains consistent across asynchronous workers.
References
- Target branch MR !243182 (merged)
- Issue #598839
- Epic gitlab-org#21890
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.