Fix race condition between policy evaluation and report ingestion

What does this MR do and why?

Fix race condition between policy evaluation and report ingestion

ProcessPipelineCompletionWorker evaluates approval policies on PipelineFinishedEvent, which may fire before Security::Finding rows are written by StoreScansService.

This causes UpdateApprovalsService to find zero findings and conclude "no violations", temporarily allowing MRs onto the merge train. When the merge train pipeline later completes and re-evaluates with findings present, the MR is removed from the train.

Introduce Security::ReportsIngestedEvent, published by StoreScansService after all findings are written. Subscribe ProcessPipelineCompletionWorker to this event so that policy evaluation runs with findings guaranteed to be present. Gate the PipelineFinishedEvent path on Security::Scan.results_ready? to skip premature evaluation when scans are still being ingested.

References

• https://gitlab.com/gitlab-com/request-for-help/-/work_items/4232

Screenshots or screen recordings

Recordings are utilizing sleep for StoreScansWorker for reliable reproduction.

Before	After

How to set up and validate locally

1. In rails console enable the feature flag

   Feature.enable(:wait_for_reports_ingestion_before_policy_evaluation)

2. Apply the patch to simulate the race condition:

   diff --git a/ee/app/workers/security/store_scans_worker.rb b/ee/app/workers/security/store_scans_worker.rb
   index 0d66b0c00a634..1ee24a9a27e8e 100644
   --- a/ee/app/workers/security/store_scans_worker.rb
   +++ b/ee/app/workers/security/store_scans_worker.rb
   @@ -13,6 +13,7 @@ class StoreScansWorker # rubocop:disable Scalability/IdempotentWorker
        feature_category :vulnerability_management
    
        def perform(pipeline_id)
   +      sleep 30
          ::Ci::Pipeline.find_by_id(pipeline_id).try do |pipeline|
            break unless pipeline.project.can_store_security_reports?
    
   

3. Configure merge trains in a project

4. Configure .gitlab-ci.yml:

   include:
     - template: Jobs/Secret-Detection.gitlab-ci.yml
   
   test:
     stage: test
     script: sleep 60
     rules:
       - if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE == "merge_train"
         when: always

5. Configure a policy:

   approval_policy:
     - name: Test policy
       description: ''
       enabled: true
       actions:
         - type: require_approval
           approvals_required: 1
           role_approvers:
             - owner
             - maintainer
       rules:
         - type: scan_finding
           branches: []
           scanners:
             - secret_detection
           vulnerabilities_allowed: 0
           severity_levels:
             - critical
             - high
             - medium
             - unknown
           vulnerability_states:
             - new_needs_triage
       approval_settings:
         block_branch_modification: false
         prevent_pushing_and_force_pushing: false
         prevent_approval_by_author: false
         prevent_approval_by_commit_author: false
         remove_approvals_with_new_commit: false
         require_password_to_approve: false
   

6. Create a merge request with a leaked secret and verify that it cannot be added to the merge train until scans are stored

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #594864