Skip approval sync for projects without approval policies
What does this MR do and why?
This MR adds two gates that skip approval sync work for projects without approval rules. Both sit behind one flag, skip_approval_sync_without_approval_policies (gitlab_com_derisk, default off), so they roll out together. The first gate is in SyncFindingsToApprovalRulesService, the second in Ci::SyncReportsToReportApprovalRulesWorker.
Gate 1: merge request approval policy sync
Security::ScanResultPolicies::ProcessPipelineCompletionWorker runs on every finished pipeline and every security report ingestion in a licensed project. It calls SyncFindingsToApprovalRulesService, which enqueues one SyncMergeRequestApprovalsWorker for each open merge request at the pipeline's head SHA, and, when the pipeline is the latest on its ref, one more for each open merge request that targets that ref. It does this even when no approval policy applies, so a busy target branch with many open merge requests produces a large number of jobs that do nothing.
This MR skips those enqueues when the merge request's target project has no approval rules created by an approval policy. Behind the skip_approval_sync_without_approval_policies flag (gitlab_com_derisk, default off). The flag actor is the merge request's target project, the same project the predicate checks.
- Head SHA merge requests: each one is checked against its own target project. For a fork merge request, the pipeline runs in the fork, but the approval rules belong to the target project. Gating on the pipeline's project would skip those merge requests. The flag is also evaluated against the target project, so enabling it for an upstream project covers pipelines that run in its forks.
- Merge requests targeting the pipeline ref, and the agentic analyzer path: these merge requests target the pipeline's project, so that project is the predicate subject and the flag actor, checked once.
- Predicate: taken from
UnenforceablePolicyRulesPipelineNotificationWorkerand moved toSecurity::ScanResultPolicies::ApprovalPolicyApplicability.applicable?so both callers share it. It is true when a policy feature is available and the project has an approval rule linked to a policy (byapproval_policy_rule_idor the legacyscan_result_policy_id).UpdateApprovalsServiceonly evaluates the merge request'sscan_findingrules, which are copied from these project rules. So a skip either leaves nothing to evaluate or leaves the existing violation in place, and the merge request stays blocked. - Query cost: the predicate is one
EXISTSquery. The result is not cached between runs. Within one run it is memoized (with the flag check) per target project, and target projects are already preloaded, so the number of queries does not depend on how many merge requests there are.
Trade-off: interrupted legacy delete. The gate reads the project's rules, but the skipped work reads the merge request's rules. The legacy ScanResultPolicy#delete_scan_finding_rules deletes project rules before merge request rules. If that delete is interrupted, a merge request can keep stale scan_finding rules that are no longer re-evaluated, so it stays blocked when it would have been unblocked. It is never unblocked by the skip, so this costs liveness, not security. Security::Policy deletes merge request rules first and is not affected.
A policy added after the pipeline finished. Linking an approval policy creates the project approval rules first, then runs SyncMergeRequestsService for every open merge request. That service schedules SyncFindingsToApprovalRulesWorker for each head pipeline (data_consistency :always), which calls this same service with the rules already in place. Merge requests skipped earlier are therefore evaluated when the policy arrives. The re-sync is batched and delayed (30 seconds, plus 30 seconds per 100 pipelines), and the deferred link path goes through SyncMergeRequestsForProjectWorker.
Gate 2: report approval rule sync
Ci::SyncReportsToReportApprovalRulesWorker is enqueued after every completed pipeline on EE, and it was the main contributor to a recent low-urgency-ci-pipeline saturation incident (a backlog of about 417k jobs). Most of those jobs have nothing to do. The service only updates license scanning and code coverage merge request approval rules. The follow-up Ci::UpdateApprovalRulesForRelatedMrsWorker already returns early when the project has no merge request approval rules.
This gate adds the same kind of early return to the first worker:
- When the project has no rows in
approval_merge_request_rules, the worker returns without running the service or enqueuing the follow-up worker. The check is a singleEXISTSon the indexedapproval_merge_request_rules.project_idcolumn. That column always holds the merge request's target project. - Projects in a fork network always run. Their pipelines can touch merge requests that target another project in the network, so the pipeline's project alone can't tell whether rules exist. This keeps fork merge requests that target projects with policies fully evaluated.
Nothing else is skipped. The only other side effect, pipeline.update_builds_coverage, is a catch-up for Ci::BuildFinishedWorker, which already records build coverage.
This gate reuses the flag from gate 1 instead of adding a second one. The flag actor is the pipeline's project. Turning the flag off restores the old behaviour for both gates.
Not included: the per-merge-request enqueue in Security::SecurityOrchestrationPolicies::SyncMergeRequestsService. It uses perform_async, so the existing until_executed / reschedule_once deduplication collapses merge requests that share a head pipeline into one queued job and at most one trailing run.
Related to #634512.
Database
The MR adds three read queries and replaces two others. All are bounded LIMIT 1 / EXISTS lookups. Plans come from production-size data on postgres.ai (first run on a fresh clone is cold; the rerun is warm). Ids below are placeholders; the console links hold the real query and require a postgres.ai session.
1. Approval policy rules on the project (gate 1, replaces two queries)
Security::ScanResultPolicies::ApprovalPolicyApplicability.applicable? replaces two .any? calls on approval_project_rules (the old UnenforceablePolicyRulesPipelineNotificationWorker check) with one query. It also runs from SyncFindingsToApprovalRulesService now, memoized per target project within a run, and only when the feature flag is on.
Before (two queries, the second only if the first finds nothing):
SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."approval_policy_rule_id" IS NOT NULL LIMIT 1
SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."scan_result_policy_id" IS NOT NULL LIMIT 1After (one query, scoped to report_approver in 9986c0b3 after @reza-marandi's review):
SELECT 1 AS one FROM "approval_project_rules" WHERE "approval_project_rules"."project_id" = <project_id> AND "approval_project_rules"."rule_type" = 2 AND ("approval_project_rules"."approval_policy_rule_id" IS NOT NULL OR "approval_project_rules"."scan_result_policy_id" IS NOT NULL) LIMIT 1Policy-linked rules are always report_approver (rule_type 2, set by ApprovalRuleParamsBuilder), and production has no policy-linked rules of another type. So the shared predicate, which UnenforceablePolicyRulesPipelineNotificationWorker also uses, now ignores policy-linked rules that are not report_approver, and in practice there are none. The scope puts rule_type in the index condition of index_approval_project_rules_on_project_id_and_rule_type. Plans of the final query, from @reza-marandi's review:
| Case | Buffers | Plan |
|---|---|---|
| Final query, project with a policy rule | 7 | plan |
| Final query, project without a policy rule | 6 | plan |
Previous query (no rule_type), project with a policy rule |
284 | plan |
Earlier plans, measured before the rule_type scope: all use an Index Scan on index_approval_project_rules_on_project_id_and_rule_type (project_id condition, the two policy columns as a filter). The table holds a few hundred rules at most per project, so the filter scan stays small. Both projects below are among the largest by rule count.
| Case | Queries | Buffers | Cold | Warm | Plan |
|---|---|---|---|---|---|
| After, project with a policy rule | 1 | 284 read | 195 ms | 0.4 ms | cold, warm |
| After, project without a policy rule | 1 | 80 (1 hit, 79 read) | 45 ms | 0.14 ms | cold, warm |
| Before, project with a policy rule | 1 | 284 read | 198 ms | - | cold |
| Before, project without a policy rule | 2 | 79 read + 80 hit | 43 ms + 0.1 ms | - | first, second |
The second old query ran after the first had cached the same pages, so its time is a warm figure. I did not get a clean warm rerun of the old queries on the shared clone and left them out.
2. Merge request approval rules on the project (gate 2, new)
Ci::SyncReportsToReportApprovalRulesWorker runs this before doing any work, only with the flag on and outside a fork network.
SELECT 1 AS one FROM "approval_merge_request_rules" WHERE "approval_merge_request_rules"."project_id" = <project_id> LIMIT 1Index Only Scan on index_approval_merge_request_rules_on_project_id; at most 8 buffers either way, so the size of the table does not matter here.
| Case | Buffers | Cold | Warm | Plan |
|---|---|---|---|---|
| Project with rules (returns a row) | 8 (3 hit, 5 read) | 5.4 ms | 0.07 ms | cold, warm |
| Project without rules (no row) | 7 (4 hit, 3 read) | 2.1 ms | 0.05 ms | cold, warm |
3. Fork network membership of the pipeline project (gate 2, new)
The same worker checks project.fork_network_member (a has_one) so fork networks always run.
SELECT "fork_network_members".* FROM "fork_network_members" WHERE "fork_network_members"."project_id" = <project_id> LIMIT 1Index Scan on the unique index_fork_network_members_on_project_id.
| Case | Buffers | Cold | Warm | Plan |
|---|---|---|---|---|
| Fork network member | 4 read | 3.6 ms | 0.02 ms | cold, warm |
| Not a member | 3 (1 hit, 2 read) | 2.2 ms | 0.02 ms | cold, warm |
References
- References: tracking issue to be created
- Gate 2 relates to #634512
Rollout notes
- One flag covers both gates.
- Gate 1: the flag actor is the merge request's target project (a
Project), so enabling an upstream project also covers merge requests from its forks. A fork's own merge requests follow the fork's flag state. - Gate 2: the flag actor is the pipeline's project.
How to set up and validate locally
- Enable the flag:
Feature.enable(:skip_approval_sync_without_approval_policies). - In a project without approval policies, open several merge requests that target the default branch, each with a head pipeline.
- Run a pipeline on the default branch and let it finish.
- Check Sidekiq: no
Security::ScanResultPolicies::SyncMergeRequestApprovalsWorkerjobs are enqueued for these merge requests. - Add an approval policy (with
scan_findingrules) to the project and repeat step 3: one job is enqueued for each open merge request again. - Fork the project, open a merge request from the fork to the project that has the policy, and run a pipeline in the fork: the merge request still gets a job.
Gate 2 (report approval rule sync), with the same flag enabled:
- In a project without merge request approval rules and outside any fork network, run
Ci::SyncReportsToReportApprovalRulesWorker.new.perform(pipeline.id)for one of its pipelines (withSidekiq::Testing.fake!). NoCi::UpdateApprovalRulesForRelatedMrsWorkerjob is enqueued. - Add a license scanning approval policy, open a merge request, and run a pipeline. The policy rule is still evaluated after the pipeline completes.
- Run
Feature.disable(:skip_approval_sync_without_approval_policies)and repeat step 1. The follow-up worker is enqueued again.
Specs:
bin/rspec ee/spec/services/security/scan_result_policies/sync_findings_to_approval_rules_service_spec.rb \
ee/spec/lib/security/scan_result_policies/approval_policy_applicability_spec.rb \
ee/spec/workers/security/unenforceable_policy_rules_pipeline_notification_worker_spec.rb \
ee/spec/workers/security/scan_result_policies/process_pipeline_completion_worker_spec.rb \
ee/spec/workers/security/scan_result_policies/sync_findings_to_approval_rules_worker_spec.rb \
ee/spec/workers/ci/sync_reports_to_report_approval_rules_worker_spec.rbMR acceptance checklist
Evaluate this MR against the MR acceptance checklist.