Make forced security policy resync state-based

What does this MR do and why?

A forced security policy resync (the resyncSecurityPolicies mutation, reassigning a policy project, or a CSP change) enqueues one SyncProjectPolicyWorker per project and per policy. Each in-scope job then sweeps the project's open merge requests twice (unlink, then link). Those jobs are enqueued with perform_in, and Sidekiq deduplication skips scheduled jobs unless the worker sets including_scheduled (deduplicates_when_scheduling.rb:60). SyncProjectPolicyWorker does not set it, so a second resync stacks a full new set of P × K jobs instead of coalescing with the pending one. On GitLab.com, one policy-project merge peaked at about 18.5k SyncProjectPolicyWorker jobs per minute (187k per hour), and the same set of about 4.2k projects was re-synced 158 times in one week.

Behind the state_based_security_policy_resync flag (gitlab_com_derisk, off by default):

  • One job per project. A forced resync goes through the same configuration drip that assignment uses: SyncPolicyConfigurationWorker → one SyncProjectPoliciesWorker per project. Repeated resyncs cost P jobs instead of P × K.
  • No overlapping runs. SyncProjectPoliciesWorker now uses until_executed with including_scheduled and reschedule_once. A duplicate that arrives while a job runs becomes one trailing run, so two unlink-and-relink passes never run at the same time for one project. Approval rule creation is check-then-create, so overlapping passes could otherwise create duplicate project rules. This dedup change is not behind the flag. It is safe for the assignment path too: the job reads current state, so re-running it with its own arguments is idempotent.
  • Same repair as before. LinkPolicyConfigurationService gets a force_resync mode. For every policy of the configuration it runs the same unlink-then-link as the per-policy resync: stale rules of unchanged policies are rebuilt, and out-of-scope and disabled policies are cleaned up.
  • One sweep per project. The merge request sweep is deferred to a single SyncMergeRequestsForProjectWorker per project.
    • Rule deletion still removes merge request rules and violations inline (Security::Policy#delete_approval_policy_rules_for_project), so no merge request keeps rules from an unlinked policy.
    • Until the sweep runs, merge requests that lack a re-created rule report checking and block merging (fail-closed).
  • Configuration-level syncs unchanged. The compliance framework and pipeline execution policy metadata syncs still run once per policy, from SyncPolicyWorker. Its per-policy project fan-out is skipped through a skip_project_sync marker on the resync event.
  • DB-health deferral. SyncProjectPolicyWorker and SyncProjectPoliciesWorker defer for 1 minute on database health signals for approval_merge_request_rules, approval_project_rules and scan_result_policy_violations. Deferral only delays the work.
    • The same flag turns this on for every job these two workers run, not only resyncs: policy create and update diffs, compliance framework, protected branch, default branch and security attribute events, and assignment.
    • It ships with the state-based resync because a deferred job is re-enqueued without deduplication. With the old per-policy resync, every deferral during a database incident would add duplicates to a P × K backlog. With one coalescing job per project, each new trigger during the incident adds at most one job per project.

The flag is read once in PersistSecurityPoliciesWorker, so the per-policy events and the configuration event always agree on which worker syncs the projects. With the flag off, the previous path runs unchanged.

Rollout: enable state_based_security_policy_resync only after this change is deployed on every node. Older Sidekiq nodes reject the new force_resync field in PolicyConfigurationAssignedEvent, because its schema disallows additional properties, and retry the job until a new node picks it up. The rollout issue must repeat this.

Related: !264010 (merged) (rate limit on resyncSecurityPolicies) and !264011 (merged) (debounce of SyncScanPoliciesWorker).

How to set up and validate locally

  1. Enable the flag: Feature.enable(:state_based_security_policy_resync).

  2. Use a group with a security policy project, a few projects, and at least two enabled approval policies, plus an open merge request in one of the projects.

  3. In the Rails console, edit one project approval rule that comes from a policy (for example, change approvals_required).

  4. Run the resync mutation for the group twice:

    mutation { resyncSecurityPolicies(input: { fullPath: "my-top-group" }) { errors } }
  5. Check that:

    • Sidekiq logs show one Security::SyncProjectPoliciesWorker per project, one Security::ScanResultPolicies::SyncMergeRequestsForProjectWorker per project with open merge requests, and no Security::SyncProjectPolicyWorker jobs.
    • The edited approval rule is re-created with the value from the policy.
  6. Disable the flag and repeat. The old per-policy fan-out runs.

Specs:

bin/rspec ee/spec/services/security/security_orchestration_policies/link_policy_configuration_service_spec.rb \
  ee/spec/workers/security/sync_policy_configuration_worker_spec.rb \
  ee/spec/workers/security/sync_project_policies_worker_spec.rb \
  ee/spec/workers/security/sync_project_policy_worker_spec.rb \
  ee/spec/workers/security/sync_policy_worker_spec.rb \
  ee/spec/workers/security/persist_security_policies_worker_spec.rb

Database

LinkPolicyConfigurationService#resyncable_policies now selects the policies to resync in one query, using the new Security::Policy.linked_to_project scope. It replaces one exists? query per disabled policy. It runs once per project job in a forced resync. The result set is bounded by the policies of one configuration.

SELECT "security_policies".*
FROM "security_policies"
WHERE "security_policies"."security_orchestration_policy_configuration_id" = <configuration_id>
  AND (policy_index >= 0)
  AND ("security_policies"."enabled" = TRUE
    OR "security_policies"."type" = 0
    OR "security_policies"."id" IN (
      SELECT "security_policy_project_links"."security_policy_id"
      FROM "security_policy_project_links"
      WHERE "security_policy_project_links"."project_id" = <project_id>));

Plans from a Database Lab clone of gitlab-production-main, using a large configuration and a project with links (EXPLAIN (ANALYZE, BUFFERS)):

Index Scan using idx_security_policies_config_id_policy_index on security_policies
  Index Cond: (security_orchestration_policy_configuration_id = <configuration_id> AND policy_index >= 0)
  Filter: (enabled OR (type = 0) OR (id = ANY (hashed SubPlan 1)))
  SubPlan 1
    -> Index Scan using idx_security_policy_project_links_on_project_id_and_id on security_policy_project_links
         Index Cond: (project_id = <project_id>)
Case Cache Execution Buffers
Configuration with enabled or approval policies only cold (fresh clone) 1.1 ms shared hit=8 read=23
Same warm 1.4 ms shared hit=30 read=1
Configuration with disabled non-approval policies cold 18.7 ms shared hit=29 read=16
Same warm 0.24 ms shared hit=45
  • Both cases use index scans only, with no sequential scan or sort.
  • The link sub-select is a hashed SubPlan that runs at most once per query. In the first case it is never executed, because every policy already matches enabled or type.
  • The cold-case time is dominated by reads of the project's link index.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading