Trigger SAST FP + VR detection for all existing vulnerabilities
## Problem to Solve
Currently, the SAST FP detection workflow only triggers for brand new vulnerabilities created during a pipeline run (via `after_create_commit` in MR gitlab!208276). This means:
- Existing vulnerabilities in the project are never analyzed for false positives
- Customers with vulnerability backlogs must wait for new detections to benefit from the feature
- Running a new pipeline doesn't re-analyze existing High/Critical vulnerabilities
This significantly limits the feature's value for customers with existing security debt.
## Decision
Similarly to https://gitlab.com/groups/gitlab-org/-/work_items/21734+:
* In the vulnerability report, users should be able to bulk-select findings to run the flow on.
* Users would alternatively be able to utilise a "Select All" button to run the flow on the entire backlog.
* Once a job is running, we should present the user with a progress bar/indicator in the vulnerability report page.
* Once a job is running, we should also allow the user to cancel/terminate the job at will.
## Proposed Implementation Approach
<details>
<summary>Click to expand</summary>
Add a button to the Vulnerability Report banner that:
* Queues FP detection workflow for all qualifying vulnerabilities
* Create separate worker for "backfills" as to not block "global" worker for all customers
* Provide user feedback/progress indicator
* Allow user to cancel backfill operation
</details>
## Initial Behavior
<details>
<summary>Click to expand</summary>
From MR gitlab!208276:
```ruby
# ee/app/models/ee/vulnerability.rb
after_create_commit :trigger_false_positive_detection, if: :sast?
def trigger_false_positive_detection
return unless ::Feature.enabled?(:enable_vulnerability_fp_detection, group)
::Vulnerabilities::TriggerFalsePositiveDetectionWorkflowWorker.perform_async(id)
end
```
This only fires when a **new** vulnerability record is created.
</details>
## Proposed Behavior
<details>
<summary>Click to expand</summary>
When the user clicks the button in the banner:
1. Identify all High/Critical SAST vulnerabilities in the project
2. Trigger FP detection workflow for each vulnerability
3. Respect concurrency limits to prevent resource exhaustion
4. Provide feedback to the user about the processing
</details>
## Initial Implementation Considerations
<details>
<summary>Click to expand</summary>
```ruby
# Potential implementation
class Vulnerabilities::TriggerBulkFpDetectionWorker
def perform(project_id, user_id)
project = Project.find(project_id)
# Find all High/Critical SAST vulnerabilities
vulnerabilities = project.vulnerabilities
.sast
.with_severity([:critical, :high])
.not_recently_analyzed(48.hours)
# Trigger FP detection with rate limiting
vulnerabilities.find_each do |vulnerability|
Vulnerabilities::TriggerFalsePositiveDetectionWorkflowWorker.perform_async(vulnerability.id)
end
end
end
```
</details>
## Final High-Level Architecture
The bulk execution architecture is shared across vulnerability Duo workflows, with execution state and atomic transitions coordinated in Redis.
```mermaid id="6hw2fh"
flowchart TB
subgraph MAIN[" "]
direction LR
UI["Vulnerability Report<br/>Select / Select all"]
API["GraphQL API<br/>Start · Cancel · Progress"]
START["BulkDuoWorkflow<br/>StartService"]
REGISTRY["BulkDuoWorkflowRegistry"]
subgraph TRACKABLE["WorkflowTrackable"]
direction LR
FP["SAST FP Detection<br/>WorkflowWorker"]
VR["SAST Resolution<br/>WorkflowWorker"]
FUTURE["Future Workflow<br/>Workers"]
end
DUO["Duo Workflow<br/>Service"]
UI --> API
API --> START
START --> REGISTRY
REGISTRY --> FP
REGISTRY --> VR
REGISTRY -.-> FUTURE
FP --> DUO
VR --> DUO
FUTURE -.-> DUO
end
STATE[("Redis Execution State<br/>Execution · Stages · Items · Progress")]
LUA["Lua Scripts<br/>Atomic transitions · Batch coordination"]
START -->|"Start + first batch"| STATE
FP -->|"Complete / fail / cancel"| STATE
VR -->|"Complete / fail / cancel"| STATE
STATE -->|"Next severity-ordered batch"| REGISTRY
STATE -.->|"Progress / status"| API
STATE <--> LUA
classDef frontend fill:#e9d7ff,stroke:#7759c2,stroke-width:2px,color:#171321
classDef api fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#172033
classDef orchestration fill:#fff1d6,stroke:#c17d10,stroke-width:2px,color:#33250f
classDef state fill:#dcfce7,stroke:#238636,stroke-width:2px,color:#14291a
classDef workflow fill:#fce7f3,stroke:#b83280,stroke-width:2px,color:#331525
classDef future fill:#f3f4f6,stroke:#6b7280,stroke-width:1.5px,color:#292d32
classDef duo fill:#ffe4d6,stroke:#e24329,stroke-width:2px,color:#331811
class UI frontend
class API api
class START,REGISTRY orchestration
class STATE,LUA state
class FP,VR workflow
class FUTURE future
class DUO duo
```
`StartService` starts the execution and schedules the first batch through `BulkDuoWorkflowRegistry`. Workflow workers are wrapped by `WorkflowTrackable`, which records completion, failure or cancellation and advances the next severity-ordered batch.
Execution state is stored in Redis, with Lua scripts used for atomic state transitions and batch coordination. The same orchestration supports SAST FP Detection and SAST Vulnerability Resolution and can be reused by future vulnerability Duo workflows.
The Redis execution state is workflow-agnostic. It tracks generic execution items rather than vulnerability-specific state, allowing the same orchestration model to support other workflows while tracking their execution state and progress.
<details>
<summary>Click to expand</summary>
```mermaid id="nsv7ne"
flowchart LR
EXEC["Execution State<br/>Status · Current stage · Progress"]
LUA["Lua Scripts<br/>Atomic transitions<br/>Batch coordination"]
ITEMS["Item State<br/>Pending · Processing<br/>Completed · Failed · Cancelled"]
EXEC <--> LUA
LUA <--> ITEMS
classDef state fill:#dcfce7,stroke:#238636,stroke-width:2px,color:#14291a
classDef lua fill:#fff1d6,stroke:#c17d10,stroke-width:2px,color:#33250f
class EXEC,ITEMS state
class LUA lua
```
The execution state tracks the lifecycle, stage, and progress of a bulk workflow execution, while item state tracks the individual units of work being processed. Lua scripts provide atomic operations for claiming batches and transitioning state safely across concurrent workers.
The orchestration and Redis state model are not coupled to vulnerabilities. Vulnerability findings are the current item type used by SAST FP Detection and SAST Vulnerability Resolution, but the same execution infrastructure can be reused by future bulk workflows with different item types.
</details>
## Final Implementation Approach
Add bulk workflow execution to the Vulnerability Report that:
* Allows users to select findings or run the workflow across the entire eligible backlog.
* Uses the shared Bulk Workflow orchestration to execute both SAST FP Detection and SAST Vulnerability Resolution.
* Processes findings in severity order with per-execution and concurrency limits.
* Tracks execution and item state in Redis.
* Uses the existing workflow workers through `WorkflowTrackable`.
* Provides a progress/status indicator in the Vulnerability Report.
* Allows users to cancel an active execution.
* Allows the same orchestration infrastructure to be reused by other and future workflows.
## Related Issues
- Parent Epic: https://gitlab.com/groups/gitlab-org/-/work_items/20432
- Related: gitlab#581652 (Banner implementation)
- Related: gitlab#581975 (Manual trigger for single vulnerability)
- Related: https://gitlab.com/groups/gitlab-org/-/epics/18977#note_2904564049
- Related: https://gitlab.com/groups/gitlab-org/-/epics/19897 (Event-based triggers architecture)
## Questions to Resolve
- [x] Should the button be available only once or multiple times?
- Multiple times. Users can start the workflow again after the current execution reaches a terminal state. Only one active execution per project/workflow is allowed at a time.
- [x] How do we handle projects with hundreds of existing vulnerabilities?
- Findings are processed in severity order and in bounded batches rather than enqueueing the entire backlog at once. Two limits apply: a per-execution limit on the total number of items and a concurrency limit on how many items can be processed concurrently.
- [x] What feedback should we provide to users during processing?
- A progress indicator in the Vulnerability Report showing the execution status and processing progress, with the option to cancel an active execution.
- [x] What's the impact on Duo Workflow Service capacity?
- Load is controlled through the per-execution and concurrency limits. Work is released in bounded, severity-ordered batches rather than dispatching the entire backlog simultaneously.
- [x] Should we add a cooldown period between bulk processing requests?
- No. A single active execution per project/workflow prevents concurrent executions of the same workflow. A new execution can be started once the current execution reaches a terminal state.
epic