Sync policy updates only to projects in a project list scope
What does this MR do and why?
Every security policy update enqueues one Security::SyncProjectPolicyWorker job per project of the policy configuration. When a policy is scoped to a list of projects, the job is a no-op for every project outside that list.
On GitLab.com, a one-project edit to a project-list scope in a large group enqueued a job for every project of the group and loaded the main database.
This MR enqueues only the listed projects, plus the projects already linked to the policy, when both the old and the new scope include an explicit project list and match_mode is not any. All other scopes keep the full fan-out. The change is behind the sync_policy_for_scoped_projects_only flag (gitlab_com_derisk, default off).
Reviewable LOC: 33 (plus 114 in tests)
References
- Related to #607604
- Feature flag rollout: #634967 (closed)
- Earlier fixes for the same incident: !261734 (merged) (skip unlink for never-linked projects) and !261748 (merged) (scope the merge request rule delete to the project). With those, each extra job is cheap. This MR removes the extra jobs.
Screenshots or screen recordings
Not applicable, no UI change.
How to set up and validate locally
- Enable the flag in a Rails console:
Feature.enable(:sync_policy_for_scoped_projects_only) - In a group with several projects, create an approval policy whose
policy_scopehasprojects: { including: [...] }with two of the projects. - Add a third project to the list and merge the policy change.
- In
log/sidekiq.log, check thatSecurity::SyncProjectPolicyWorkerran only for the three listed projects, not for every project of the group.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Context for LLM agents
- Why skipping is safe:
PolicyScopeCheckerrequires every defined scope condition to match unlessmatch_modeisany. With a non-emptyprojects.including, only the listed projects can be in scope. A project listed in neither the old nor the new scope is out of scope both times.SyncProjectServicereturns without changes for it: it is not linked (unlinking it is already skipped since !261734 (merged)), and it is not linked again. - It applies to every update, not only scope changes. When the scope is unchanged, the old scope is
policy.scope. Rules-only and status-only edits both have specs. A rules or status change on a project-list policy also used to fan out to the whole group. - The candidate ids are intersected with
all_project_ids, so a listed project outside the configuration is never enqueued. The id plucking over the configuration stays. That is cheap next to enqueuing one job per project. - Rejected: syncing only projects whose scope status changed (symmetric difference). It skips relinking projects that stay in scope, which this MR does not do. The union stays close to the list size.
- Rejected: narrowing group or compliance framework scopes. Their in-scope set needs a per-project membership check. They keep the full fan-out.
- Not covered: an unlisted, unlinked project that still has approval rules from an interrupted unlink. The interrupted
SyncProjectPolicyWorkerjob retries itself, andapproval_rules_remain_in_project?handles that retry. - Only the job fan-out shrinks.
all_project_idsstill walks the configuration's project ids (the whole organization for a centralized security policy (CSP) configuration), so listed ids stay inside the configuration. - A non-any scope that combines
projects.includingwith groups or frameworks is filtered too. That is correct because every condition must match. - Create and resync events keep the full fan-out. Resync is the repair path.
- Linked projects are added to the candidates so stale links (for example from old transfers) still get cleaned up on disable or unscope. Resync skips disabled policies, so it cannot be the cleanup path.
policy.project_idsreads links bysecurity_policy_idthrough the(security_policy_id, project_id)unique index.
Owed elsewhere
- Group and compliance framework scope edits still fan out to every project. That is the case #607604 describes, so the issue stays open for it.
Database
No migrations and no new indexes. With sync_policy_for_scoped_projects_only off nothing changes. With it on, the MR adds one query, Security::Policy#linked_project_ids, run once per all_project_ids batch for a policy scoped to a project list (non-any). It reads linked projects from security_policy_project_links only. The previous policy.project_ids went through the projects association and joined projects for a column the links table already has. The limit is exact: (security_policy_id, project_id) is unique, so a batch of N ids matches at most N links.
SELECT "security_policy_project_links"."project_id"
FROM "security_policy_project_links"
WHERE "security_policy_project_links"."security_policy_id" = <policy_id>
AND "security_policy_project_links"."project_id" IN (<project_id_1>, ..., <project_id_N>)
LIMIT NPlans from postgres.ai (gitlab-production-main thin clone, ids replaced by placeholders). Batch of 1000 ids, policy with ~1e5 links.
Warm, all 1000 ids linked to the policy (largest policy)
Limit (cost=0.43..1929.00 rows=116 width=8) (actual time=0.037..0.239 rows=1000 loops=1)
Buffers: shared hit=47
-> Index Only Scan using index_security_policy_project_links_on_project_and_policy on security_policy_project_links (cost=0.43..1929.00 rows=116 width=8) (actual time=0.036..0.171 rows=1000 loops=1)
Heap Fetches: 64
Buffers: shared hit=47
Planning Time: 0.703 ms
Execution Time: 0.288 msCold (first run on a fresh clone), 1000 ids none linked to the policy (the common case: a project list is a small part of the group)
Limit (cost=0.43..1929.00 rows=116 width=8) (actual time=0.140..0.141 rows=0 loops=1)
Buffers: shared hit=3 read=3
-> Index Only Scan using index_security_policy_project_links_on_project_and_policy on security_policy_project_links (cost=0.43..1929.00 rows=116 width=8) (actual time=0.140..0.140 rows=0 loops=1)
Heap Fetches: 0
Buffers: shared hit=3 read=3
Planning Time: 2.012 ms
Execution Time: 0.160 msVerdict: index-only scan on the unique index, no join, no seq scan. Under 1 ms cold and warm at 1000 ids, bounded by the batch size. The planner estimates ~116 rows against 1000 actual, but the plan is index-driven either way.
For comparison, the previous unbounded through-association lookup for the same policy (~1e5 rows) took ~300 ms warm and ~1e6 buffer hits, because it did an Index Only Scan on projects per link row. Reading the links table alone was ~15 ms for the same rows, and the batched lookup above stays within that.
The existing all_project_ids walk is unchanged.