Skip approval sync for projects without approval policies

What does this MR do and why?

This MR adds two gates that skip approval sync work for projects without approval rules. Both sit behind one flag, skip_approval_sync_without_approval_policies (gitlab_com_derisk, default off), so they roll out together. The first gate is in SyncFindingsToApprovalRulesService, the second in Ci::SyncReportsToReportApprovalRulesWorker.

Gate 1: merge request approval policy sync

Security::ScanResultPolicies::ProcessPipelineCompletionWorker runs on every finished pipeline and every security report ingestion in a licensed project. It calls SyncFindingsToApprovalRulesService, which enqueues one SyncMergeRequestApprovalsWorker for each open merge request at the pipeline's head SHA, and, when the pipeline is the latest on its ref, one more for each open merge request that targets that ref. It does this even when no approval policy applies, so a busy target branch with many open merge requests produces a large number of jobs that do nothing.

This MR skips those enqueues when the merge request's target project has no approval rules created by an approval policy. Behind the skip_approval_sync_without_approval_policies flag (gitlab_com_derisk, default off). The flag actor is the merge request's target project, the same project the predicate checks.

  • Head SHA merge requests: each one is checked against its own target project. For a fork merge request, the pipeline runs in the fork, but the approval rules belong to the target project. Gating on the pipeline's project would skip those merge requests. The flag is also evaluated against the target project, so enabling it for an upstream project covers pipelines that run in its forks.
  • Merge requests targeting the pipeline ref, and the agentic analyzer path: these merge requests target the pipeline's project, so that project is the predicate subject and the flag actor, checked once.
  • Predicate: taken from UnenforceablePolicyRulesPipelineNotificationWorker and moved to Security::ScanResultPolicies::ApprovalPolicyApplicability.applicable? so both callers share it. It is true when a policy feature is available and the project has an approval rule linked to a policy (by approval_policy_rule_id or the legacy scan_result_policy_id). UpdateApprovalsService only evaluates the merge request's scan_finding rules, which are copied from these project rules. So a skip either leaves nothing to evaluate or leaves the existing violation in place, and the merge request stays blocked.
  • Query cost: the predicate is one EXISTS query. The result is not cached between runs. Within one run it is memoized (with the flag check) per target project, and target projects are already preloaded, so the number of queries does not depend on how many merge requests there are.

Trade-off: interrupted legacy delete. The gate reads the project's rules, but the skipped work reads the merge request's rules. The legacy ScanResultPolicy#delete_scan_finding_rules deletes project rules before merge request rules. If that delete is interrupted, a merge request can keep stale scan_finding rules that are no longer re-evaluated, so it stays blocked when it would have been unblocked. It is never unblocked by the skip, so this costs liveness, not security. Security::Policy deletes merge request rules first and is not affected.

A policy added after the pipeline finished. Linking an approval policy creates the project approval rules first, then runs SyncMergeRequestsService for every open merge request. That service schedules SyncFindingsToApprovalRulesWorker for each head pipeline (data_consistency :always), which calls this same service with the rules already in place. Merge requests skipped earlier are therefore evaluated when the policy arrives. The re-sync is batched and delayed (30 seconds, plus 30 seconds per 100 pipelines), and the deferred link path goes through SyncMergeRequestsForProjectWorker.

Gate 2: report approval rule sync

Ci::SyncReportsToReportApprovalRulesWorker is enqueued after every completed pipeline on EE, and it was the main contributor to a recent low-urgency-ci-pipeline saturation incident (a backlog of about 417k jobs). Most of those jobs have nothing to do. The service only updates license scanning and code coverage merge request approval rules. The follow-up Ci::UpdateApprovalRulesForRelatedMrsWorker already returns early when the project has no merge request approval rules.

This gate adds the same kind of early return to the first worker:

  • When the project has no rows in approval_merge_request_rules, the worker returns without running the service or enqueuing the follow-up worker. The check is a single EXISTS on the indexed approval_merge_request_rules.project_id column. That column always holds the merge request's target project.
  • Projects in a fork network always run. Their pipelines can touch merge requests that target another project in the network, so the pipeline's project alone can't tell whether rules exist. This keeps fork merge requests that target projects with policies fully evaluated.

Nothing else is skipped. The only other side effect, pipeline.update_builds_coverage, is a catch-up for Ci::BuildFinishedWorker, which already records build coverage.

This gate reuses the flag from gate 1 instead of adding a second one. The flag actor is the pipeline's project. Turning the flag off restores the old behaviour for both gates.

Not included: the per-merge-request enqueue in Security::SecurityOrchestrationPolicies::SyncMergeRequestsService. It uses perform_async, so the existing until_executed / reschedule_once deduplication collapses merge requests that share a head pipeline into one queued job and at most one trailing run.

Related to #634512.

Database

The MR adds three read queries and replaces two others. All are bounded LIMIT 1 / EXISTS lookups. Plans come from production-size data on postgres.ai (first run on a fresh clone is cold; the rerun is warm). Ids below are placeholders; the console links hold the real query and require a postgres.ai session.

1. Approval policy rules on the project (gate 1, replaces two queries)

Security::ScanResultPolicies::ApprovalPolicyApplicability.applicable? replaces two .any? calls on approval_project_rules (the old UnenforceablePolicyRulesPipelineNotificationWorker check) with one query. It also runs from SyncFindingsToApprovalRulesService now, memoized per target project within a run, and only when the feature flag is on.

Before (two queries, the second only if the first finds nothing):

SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."approval_policy_rule_id" IS NOT NULL LIMIT 1
SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."scan_result_policy_id" IS NOT NULL LIMIT 1

After (one query, scoped to report_approver in 9986c0b3 after @reza-marandi's review):

SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."rule_type" = 2 AND ("approval_project_rules"."approval_policy_rule_id" IS NOT NULL OR "approval_project_rules"."scan_result_policy_id" IS NOT NULL) LIMIT 1

Policy-linked rules are always report_approver (rule_type 2, set by ApprovalRuleParamsBuilder), and production has no policy-linked rules of another type. So the shared predicate, which UnenforceablePolicyRulesPipelineNotificationWorker also uses, now ignores policy-linked rules that are not report_approver, and in practice there are none. The scope puts rule_type in the index condition of index_approval_project_rules_on_project_id_and_rule_type. Plans of the final query, from @reza-marandi's review:

Case Buffers Plan
Final query, project with a policy rule 7 plan
Final query, project without a policy rule 6 plan
Previous query (no rule_type), project with a policy rule 284 plan

Earlier plans, measured before the rule_type scope: all use an Index Scan on index_approval_project_rules_on_project_id_and_rule_type (project_id condition, the two policy columns as a filter). The table holds a few hundred rules at most per project, so the filter scan stays small. Both projects below are among the largest by rule count.

Case Queries Buffers Cold Warm Plan
After, project with a policy rule 1 284 read 195 ms 0.4 ms cold, warm
After, project without a policy rule 1 80 (1 hit, 79 read) 45 ms 0.14 ms cold, warm
Before, project with a policy rule 1 284 read 198 ms - cold
Before, project without a policy rule 2 79 read + 80 hit 43 ms + 0.1 ms - first, second

The second old query ran after the first had cached the same pages, so its time is a warm figure. I did not get a clean warm rerun of the old queries on the shared clone and left them out.

2. Merge request approval rules on the project (gate 2, new)

Ci::SyncReportsToReportApprovalRulesWorker runs this before doing any work, only with the flag on and outside a fork network.

SELECT 1 AS one FROM "approval_merge_request_rules" WHERE "approval_merge_request_rules"."project_id" = <project_id> LIMIT 1

Index Only Scan on index_approval_merge_request_rules_on_project_id; at most 8 buffers either way, so the size of the table does not matter here.

Case Buffers Cold Warm Plan
Project with rules (returns a row) 8 (3 hit, 5 read) 5.4 ms 0.07 ms cold, warm
Project without rules (no row) 7 (4 hit, 3 read) 2.1 ms 0.05 ms cold, warm

3. Fork network membership of the pipeline project (gate 2, new)

The same worker checks project.fork_network_member (a has_one) so fork networks always run.

SELECT "fork_network_members".* FROM "fork_network_members" WHERE "fork_network_members"."project_id" = <project_id> LIMIT 1

Index Scan on the unique index_fork_network_members_on_project_id.

Case Buffers Cold Warm Plan
Fork network member 4 read 3.6 ms 0.02 ms cold, warm
Not a member 3 (1 hit, 2 read) 2.2 ms 0.02 ms cold, warm

References

  • References: tracking issue to be created
  • Gate 2 relates to #634512

Rollout notes

  • One flag covers both gates.
  • Gate 1: the flag actor is the merge request's target project (a Project), so enabling an upstream project also covers merge requests from its forks. A fork's own merge requests follow the fork's flag state.
  • Gate 2: the flag actor is the pipeline's project.

How to set up and validate locally

  1. Enable the flag: Feature.enable(:skip_approval_sync_without_approval_policies).
  2. In a project without approval policies, open several merge requests that target the default branch, each with a head pipeline.
  3. Run a pipeline on the default branch and let it finish.
  4. Check Sidekiq: no Security::ScanResultPolicies::SyncMergeRequestApprovalsWorker jobs are enqueued for these merge requests.
  5. Add an approval policy (with scan_finding rules) to the project and repeat step 3: one job is enqueued for each open merge request again.
  6. Fork the project, open a merge request from the fork to the project that has the policy, and run a pipeline in the fork: the merge request still gets a job.

Gate 2 (report approval rule sync), with the same flag enabled:

  1. In a project without merge request approval rules and outside any fork network, run Ci::SyncReportsToReportApprovalRulesWorker.new.perform(pipeline.id) for one of its pipelines (with Sidekiq::Testing.fake!). No Ci::UpdateApprovalRulesForRelatedMrsWorker job is enqueued.
  2. Add a license scanning approval policy, open a merge request, and run a pipeline. The policy rule is still evaluated after the pipeline completes.
  3. Run Feature.disable(:skip_approval_sync_without_approval_policies) and repeat step 1. The follow-up worker is enqueued again.

Specs:

bin/rspec ee/spec/services/security/scan_result_policies/sync_findings_to_approval_rules_service_spec.rb \
  ee/spec/lib/security/scan_result_policies/approval_policy_applicability_spec.rb \
  ee/spec/workers/security/unenforceable_policy_rules_pipeline_notification_worker_spec.rb \
  ee/spec/workers/security/scan_result_policies/process_pipeline_completion_worker_spec.rb \
  ee/spec/workers/security/scan_result_policies/sync_findings_to_approval_rules_worker_spec.rb \
  ee/spec/workers/ci/sync_reports_to_report_approval_rules_worker_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading