Debounce policy configuration sync after policy project changes

What does this MR do and why?

Two security policy sync paths multiply work when something triggers them in a burst. This MR debounces both, behind one gitlab_com_derisk flag, debounce_security_policy_sync. The flag was renamed from debounce_security_policy_configuration_sync when the second debounce was folded in, so the name no longer mentions only the configuration sync. Actor: the root namespace of the configuration's project or group for the configuration sync, and the project or group of the event for the protected branch sync.

Policy configuration sync

A merge into a security policy project re-syncs every project linked to each configuration that uses it. Repeated triggers stack instead of coalescing, so a burst of merges multiplies the work.

The post-merge hook and the outdated-configuration cron now call Security::SyncScanPoliciesWorker.perform_debounced. The first call for a configuration schedules the sync one minute ahead. Further calls within that minute are skipped (Gitlab::ExclusiveLease.throttle, lease 5 s shorter than the delay, so it always expires before the scheduled job runs). The sync reads the policy YAML when it runs, so it picks up every change from the burst. Policy changes take effect up to one minute later.

Why a throttle lease and not deduplicate including_scheduled: true

With until_executed + including_scheduled, the dedup key is held until the scheduled job finishes. DuplicateJob#reschedulable? is false for scheduled jobs, so a call made while the sync is executing would be dropped without a trailing run. A merge during the sync could then stay unapplied until the next cron. The lease expires before the scheduled job starts (lease period = perform_in delay minus a 5 s margin for clock skew), so every skipped call happens before the pending job reads the YAML. Calls after that schedule a new job. That job goes through the existing until_executed + reschedule_once dedup when it is enqueued. The worker's deduplicate options are unchanged.

Protected branch events

Every protected branch create or delete publishes an event that runs Security::SyncPolicyEventWorker, which enqueues one Security::SyncProjectPolicyWorker per approval policy of the project (or of every project in the group). Bots that create and delete protected branches in a loop turn each API call into a full policy resync. On GitLab.com this worker ran about 9M jobs in a week, and most of them came from a single top-level group's bots. Deduplication removed none of them.

Deduplication never fired for two reasons:

  • ProtectedBranchCreatedEvent carries a unique protected_branch_id, so no two jobs share arguments.
  • deduplicate :until_executing releases its key when the job starts. The jobs are cheap and arrive one API call apart, so each job has started before the next one is enqueued.

With the flag on, this MR:

  • Schedules protected branch events 30 seconds out, through a second, flag-gated subscription.
  • Deduplicates them on the event type and parent only (idempotency_arguments) with including_scheduled: true, so a burst for one project or group collapses into the pending job.

The resync is deferred, never dropped. The handler reads the parent's current protected branches and policies when it runs, and ignores the payload beyond the parent. An event is only deduplicated while a job for the same parent is still pending. That job's key is released when it starts, so a change made after that schedules a new run. The last change in a burst is therefore always synced. The worst-case extra latency is the 30-second delay.

idempotency_arguments does not depend on the flag, so a flag flip cannot leave a key behind that no job releases. With the flag off, jobs are enqueued immediately as before, but the narrower dedup key still applies: created events for a parent that are queued at the same moment now share a key and collapse, as destroyed events already did. That is safe because the handler ignores the payload beyond the parent and reads current state when it runs. A job that is still queued will read the same state as the one it replaces, and the key is released when a job starts, so any later event still enqueues a new run.

This part does not overlap with !263995 (merged), which changes SyncPolicyEventService.

What this does not change
  • Assign, reassign, user resync and compliance/security policy setting changes still enqueue immediately. Reassign force-resyncs the same configuration, so delaying it would leave a new policy project unenforced for longer.
  • No job is dropped. Calls are delayed and coalesced only.
  • The per-project fan-out and its pacing are unchanged; the per-class concurrency limit still bounds execution.

References

  • Tracking issue: to be created.
  • Related: !263738 (merged) (narrows fan-out for project-list scopes)

Database

No new tables, columns or indexes. The only query change is the new scope Security::OrchestrationPolicyConfiguration.with_source_namespace (includes(:namespace, project: :namespace)). It is a pure preload: it adds no WHERE or JOIN to the configuration query, only batched primary key lookups for the rows that query already returned. The debounce flag check reads the configuration's root namespace, which previously loaded the project and namespace once per configuration (N+1).

SQL

Post-merge hook (EE::MergeRequests::PostMergeService, one query per merge into a policy project):

SELECT "security_orchestration_policy_configurations".* FROM "security_orchestration_policy_configurations" WHERE "security_orchestration_policy_configurations"."security_policy_management_project_id" = $1;
SELECT "projects".<columns> FROM "projects" WHERE "projects"."id" IN ($1, $2, ...);
SELECT "namespaces".<columns> FROM "namespaces" WHERE "namespaces"."id" IN ($1, $2, ...);   -- configuration.namespace
SELECT "namespaces".<columns> FROM "namespaces" WHERE "namespaces"."id" IN ($1, $2, ...);   -- project.namespace

Outdated-configuration cron (Security::CreateOrchestrationPolicyWorker, per each_batch batch). The batching query is unchanged; with_outdated_configuration already had the JOIN:

SELECT "security_orchestration_policy_configurations"."id" FROM "security_orchestration_policy_configurations" INNER JOIN "projects" ON "projects"."id" = "security_orchestration_policy_configurations"."security_policy_management_project_id" WHERE ("security_orchestration_policy_configurations"."configured_at" < "projects"."last_repository_updated_at" OR "security_orchestration_policy_configurations"."configured_at" IS NULL) ORDER BY "security_orchestration_policy_configurations"."id" ASC LIMIT 1;
SELECT "security_orchestration_policy_configurations".* FROM "security_orchestration_policy_configurations" INNER JOIN "projects" ON ... WHERE "security_orchestration_policy_configurations"."id" >= <start_id> AND "security_orchestration_policy_configurations"."id" <= <end_id> AND (<same WHERE as above>);
SELECT "projects".<columns> FROM "projects" WHERE "projects"."id" IN ($1, $2, ...);
SELECT "namespaces".<columns> FROM "namespaces" WHERE "namespaces"."id" IN ($1, $2, ...);   -- configuration.namespace
SELECT "namespaces".<columns> FROM "namespaces" WHERE "namespaces"."id" IN ($1, $2, ...);   -- project.namespace

<columns> stands for the full column list Rails selects. Each IN list is bounded by the configuration batch (at most 1000 ids in the cron, the configurations of one policy project in the hook).

Query plans: production-size plans from the Database Lab clone (gitlab-production-main), each run cold (after a clone reset) and then warm. The IN lists hold the ids of a policy project with over a hundred configurations. All of them use index scans (no sequential scans), and the warm runs read only shared buffers.

Query Plan Cold Warm Warm buffers Console links
Configurations of one policy project (hook) Index Scan on index_sop_configurations_project_id_policy_project_id 3.7 ms 0.9 ms 132 hit cold, warm
Same, for a policy project with over a thousand configurations same index 182 ms 1.6 ms 1,169 hit cold, warm
projects preload (about a hundred ids) Index Scan on projects_pkey 129 ms 5.0 ms 358 hit cold, warm
namespaces preload, configuration.namespace Index Scan on namespaces_pkey 35 ms 2.8 ms 45 hit cold, warm
namespaces preload, project.namespace Index Scan on namespaces_pkey 77 ms 2.7 ms 80 hit cold, warm

The cold times are I/O reads on a freshly reset clone. The configuration query and the JOIN of the cron are unchanged from master, so I did not add plans for them.

How to set up and validate locally

  1. Enable the flag:
    Feature.enable(:debounce_security_policy_sync)
  2. Link a security policy project to a group with several projects.
  3. Merge two MRs that change policy.yml within one minute. In log/sidekiq.log, there is one Security::SyncScanPoliciesWorker run about a minute after the first merge, and it applies both changes.
  4. In a project with an approval policy, create and delete several protected branches in quick succession through the API. Check that one Security::SyncPolicyEventWorker job per event type runs about 30 seconds after the last change, and that the project's approval rules match the final protected branches.
  5. Specs:
    bin/rspec ee/spec/workers/security/sync_scan_policies_worker_spec.rb \
      ee/spec/workers/security/create_orchestration_policy_worker_spec.rb \
      ee/spec/services/ee/merge_requests/post_merge_service_spec.rb \
      ee/spec/workers/security/sync_policy_event_worker_spec.rb

Test results

91 examples, 0 failures

RuboCop on the changed files: no offenses detected.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading