Resync security policies with one configuration event

What does this MR do and why?

A forced security policy resync (the resyncSecurityPolicies mutation, a policy project change, or a CSP change) publishes one PolicyResyncEvent per policy. Each event fans out one SyncProjectPolicyWorker per project, and each in-scope job sweeps the project's open merge requests. A resync of K policies over P projects costs P × K jobs and up to 2 × K merge request sweeps per project. This is the problem from #600291. It replaces !265393 (closed) with a smaller scope, following the review there.

Behind the state_based_security_policy_resync flag (gitlab_com_derisk, off by default), a forced resync works like assignment from #600265 (closed):

  • One event per request.
    • PersistSecurityPoliciesWorker publishes one PolicyConfigurationResyncEvent for the configuration. EventPublisher stops publishing a PolicyResyncEvent per policy, and checks the flag itself.
    • SyncPolicyConfigurationWorker subscribes to the new event. It runs the compliance framework sync and the pipeline execution policy metadata sync once for each enabled policy, as SyncPolicyWorker did for each resync event. Then it enqueues one SyncProjectPoliciesWorker per project.
    • It also enqueues DeleteSecurityPolicyWorker for the configuration's soft-deleted policies, in case an earlier delete job never ran.
    • Its dedup key now includes the event type, so an assignment and a resync of the same configuration don't deduplicate into each other.
  • One job and one sweep per project.
    • SyncProjectPoliciesWorker hands a forced resync to the new ResyncProjectPoliciesService. The service runs the same per-policy resync as before (SyncPolicyEventService with PolicyResyncEvent) for every undeleted policy of the configuration.
    • Applicable policies are unlinked and relinked with the merge request sweep deferred. The service then schedules one SyncMergeRequestsForProjectWorker and one finding enrichment sync for the project.
    • Disabled and out-of-scope policies unlink with the sweep deferred as well. Unlinking already deletes their merge request rules and enqueues the bot comment refresh, so a sweep after it has nothing left to do for them.
    • The project-wide scheduling helpers move from LinkPolicyConfigurationService into a small ProjectWideSyncs concern that both services share.
    • SyncProjectPoliciesWorker switches from until_executing, including_scheduled to until_executed, if_deduplicated: :reschedule_once, the same as the per-policy SyncProjectPolicyWorker. Two resyncs of one project, such as a configuration resync and an inherited one, no longer run at the same time: a duplicate enqueued while a job runs is rescheduled once after it. including_scheduled is dropped because a perform_in duplicate can't be rescheduled and would be lost; scheduled jobs are deduplicated when they become due instead.

A resync now costs P jobs instead of P × K, and one deferred merge request sweep per project instead of one for each policy. Unlike the per-policy path, which skipped disabled policies, the project job also unlinks them, so stale links are cleaned up. For a disabled pipeline execution schedule policy that enqueues one CancelPolicyPipelinesWorker per project, even where it was never linked. The worker is a no-op in that case, and the inherited resync already does the same.

The project-level resync (inherited relationship) already enqueues SyncProjectPoliciesWorker with the force resync reason, so it uses the new service too when the flag is on.

With the flag off, the previous path runs unchanged. Two changes aren't behind the flag. The SyncPolicyConfigurationWorker dedup key now includes the event type, so an assignment job enqueued before the deploy doesn't deduplicate against one enqueued after it. The SyncProjectPoliciesWorker dedup change above also applies to the assignment path: a duplicate assignment job for a project now waits for the running one and runs after it.

Left out on purpose, compared with !265393 (closed). Policy Store / Security Policies v2 (#549779 (closed)) will replace this path, so this stays small:

  • No DB-health deferral on the project policy sync workers.
  • No new query on the project fan-out. The service resyncs undeleted policies, the same set the per-policy fan-out uses. The only new query is the per-configuration deleted lookup in SyncPolicyConfigurationWorker (see Database review).
  • No force_resync field on PolicyConfigurationAssignedEvent and no skip_project_sync marker on PolicyResyncEvent.
  • No merge_request_sync_deferred? tracking.

Rollout: enable the flag only after this change is deployed on every node. An older Sidekiq node doesn't know PolicyConfigurationResyncEvent and raises InvalidEvent until a new node picks the job up. Rollout issue: #636320.

Database review

New scope Security::Policy.deleted (policy_index < 0), used once per resync in SyncPolicyConfigurationWorker:

configuration.security_policies.deleted.pluck_primary_key
SELECT "security_policies"."id" FROM "security_policies"
WHERE "security_policies"."security_orchestration_policy_configuration_id" = <configuration_id>
AND (policy_index < 0)

Plan on production-size data (gitlab-production-main, Database Lab clone, two configurations tried, same shape):

Index Scan using idx_security_policies_config_id_policy_index on security_policies
  Index Cond: ((security_orchestration_policy_configuration_id = <configuration_id>) AND (policy_index < 0))
  Buffers: shared hit=6
  execution: under 0.1 ms

Both predicates are index conditions on the existing composite index idx_security_policies_config_id_policy_index (security_orchestration_policy_configuration_id, policy_index), so no new index is needed. The query runs once per configuration resync and returns the soft-deleted rows of that one configuration, which is normally none.

The configuration-level sync also preloads security_pipeline_execution_policy_config_link for the enabled policies, which is one WHERE security_policy_id IN (...) query on the links table per configuration.

How to set up and validate locally

  1. Enable the flag: Feature.enable(:state_based_security_policy_resync).

  2. Use a group with a security policy project, a few projects, at least two enabled approval policies, and an open merge request in one project.

  3. In the Rails console, change approvals_required on one project approval rule that comes from a policy.

  4. Run the resync for the group:

    mutation { resyncSecurityPolicies(input: { fullPath: "my-top-group" }) { errors } }
  5. Check that:

    • Sidekiq logs show one Security::SyncPolicyConfigurationWorker for Security::PolicyConfigurationResyncEvent, one Security::SyncProjectPoliciesWorker per project, and one Security::ScanResultPolicies::SyncMergeRequestsForProjectWorker per project with open merge requests.
    • There are no Security::SyncProjectPolicyWorker jobs and no Security::PolicyResyncEvent jobs.
    • The edited approval rule is re-created with the value from the policy.
  6. Disable the flag and repeat. The per-policy fan-out runs.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading