Draft: PoC: track policy propagation progress and errors

What does this MR do and why?

This is a PoC for tracking of policy propagation progress.

• It uses redis to store the affected project_ids when policies are updated
• After the project approval rules are updated, projects are marked as finished
• If the project worker exhausts all retries, the projects are added into a failed_projects list to show to the user that policies were not applied correctly for these projects
• Each project sync consists of sync of open merge requests:
  • Before merge request sync starts, we add it a list
  • Before a worker responsible for some part of the MR sync starts, we increase a counter for the merge request ID
  • After the worker finishes, counter is decreased
  • If it reaches 0, the MR is removed from the sync list
• Add GQL subscription to communicate the status to the user (there's no query to get the current state in the PoC, it's subscription-only)
• Add a FE component to show the progress bar for project sync and merge request sync
• Show a list of failed project IDs if some projects fail

References

• Idea shared in #528300 (comment 2423356596)

Screenshots or screen recordings

Sync in progress	Sync failure

CleanShot_2025-07-25_at_12.42.50

How to set up and validate locally

It's best to test it with the CSP which performs a sync for all instance projects.

1. In rails console enable the feature flag

   Feature.enable(:security_policies_csp)

2. Visit Admin -> Security and Compliance and select Compliance and security policy group
3. Go to the group -> Policies and observe the sync progress
4. To test failures, apply the following patch and wait for the retries to get exhausted:

   diff --git a/ee/app/workers/security/sync_project_policy_worker.rb b/ee/app/workers/security/sync_project_policy_worker.rb
   index 5915911e1f53..0ea88991c7eb 100644
   --- a/ee/app/workers/security/sync_project_policy_worker.rb
   +++ b/ee/app/workers/security/sync_project_policy_worker.rb
   @@ -8,6 +8,7 @@ class SyncProjectPolicyWorker
    
        data_consistency :sticky
        idempotent!
   +    sidekiq_options retry: 2 # For easier testing
        deduplicate :until_executed, if_deduplicated: :reschedule_once
    
        concurrency_limit -> { 200 }
   @@ -41,6 +42,7 @@ def perform(project_id, security_policy_id, policy_changes = {}, params = {})
          security_policy = Security::Policy.find_by_id(security_policy_id)
    
          return unless project && security_policy
   +      raise "Test error" if project_id == 1
    
          if params['event'].present?
            handle_event(project, security_policy, params['event'])
   

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.