Queue group archive vulnerability jobs only for vulnerable projects

What does this MR do and why?

Archiving or unarchiving a group queues two jobs for every project in it, even projects with no vulnerabilities. Those jobs find nothing, but each takes a Sidekiq slot. On GitLab.com, 8,410 jobs came from this in 5 days. A 1,248-project subgroup queued 2,496 jobs where 0 were needed. A 736-project root group queued 1,472 where 460 were needed.

  1. Behind the new skip_group_archive_jobs_for_projects_without_vulnerabilities feature flag (off by default), jobs are queued only for projects whose has_vulnerabilities setting is on. The flag is read once per event, with the archived group as the actor.
  2. Both per-project workers get a concurrency limit of 25 and defer on the database health signal. This is not behind the flag.
  3. The worker walks groups only, skipping project namespaces.

Review notes

Alternatives considered: Filtering inside each per-project job instead of before queueing (rejected: the job count, the actual cost, would not drop). Putting the concurrency limit behind the flag too (not done: other callers of the reads worker queued 16 jobs in 5 days against 8,410 from archives, so the limit barely touches them).

Not done, tracked at: Batching the per-project jobs: second MR for the same issue https://gitlab.com/gitlab-org/gitlab/-/work_items/637720 (needs its own feature flag because the workers take one id today). The 5-minute per-project lock is shorter than the longest jobs (13 minutes), so two jobs for one project can overlap: https://gitlab.com/gitlab-org/gitlab/-/work_items/638907. Ingestion during a group archive writes archived = false because it uses the project's own archived column: pre-existing, no issue yet. Per-project route lookup when building job contexts: pre-existing, no issue yet.

  • Approvals: backend maintainer (CODEOWNERS [Maintainers] covers ee/ and ee/config/feature_flags/). No db/ files change; the Database review section below covers the one changed query.
  • Deviation from the issue: the issue did not ask for namespace_class: Group or the database health deferral; both mirror the merged sibling worker Security::AnalyzersStatus::ProcessGroupArchivedEventsWorker / UpdateArchivedAnalyzerStatusWorker.
  • Why 25: the value the sibling security workers use; locally, with the limit at 2, 111 of 113 jobs per worker were deferred and all finished, so deferred jobs are never dropped. The largest production archive ran 6-7 jobs at once, under 25.
  • Flag actor: the archived group, so the flag can be enabled for one test group inside gitlab-org. A subgroup archived on its own without the flag queues every project, which is today's behaviour.
  • Known risk, accepted: a project with vulnerabilities but has_vulnerabilities off would be skipped and keep a stale archived value. A production check of both groups found none. The rollout issue holds the check and the repair step: https://gitlab.com/gitlab-org/gitlab/-/work_items/638850
  • Accepted side effect: leftover vulnerability_statistics rows with zero counts and the setting off keep a stale archived value; the grades panel already excludes archived projects through Project.non_archived.

Database review

The only changed statement is the per-batch project load, which now joins project_settings and uses the partial index index_project_settings_on_project_id_partially.

SQL, plans and reading

One real batch on gitlab-org (9970): the 100 lowest-id projects in its tree, which sit in 5 namespaces.

Before:

SELECT "projects".* FROM "projects" WHERE "projects"."namespace_id" IN (9970, 1793858, 1920510, 2287432, 14018650) AND "projects"."id" >= 13083 AND "projects"."id" < 3605986

After:

SELECT "projects".* FROM "projects" INNER JOIN "project_settings" ON "project_settings"."project_id" = "projects"."id" WHERE "projects"."namespace_id" IN (9970, 1793858, 1920510, 2287432, 14018650) AND "projects"."id" >= 13083 AND "projects"."id" < 3605986 AND (has_vulnerabilities IS TRUE)
Statement Plan Rows Buffers (warm) Execution (warm)
Before https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82707/commands/190227 100 120 0.4 ms
After https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82707/commands/190229 30 425 0.6 ms
  • projects uses index_projects_on_namespace_id_and_id in both plans (120 buffers).
  • project_settings is an index-only scan on index_project_settings_on_project_id_partially, once per project in the batch (305 buffers). That is the whole difference.

How to set up and validate locally

bin/rspec ee/spec/workers/vulnerabilities/process_group_archived_events_worker_spec.rb ee/spec/workers/vulnerabilities/update_archived_attribute_of_vulnerability_reads_worker_spec.rb ee/spec/workers/vulnerabilities/update_archived_attribute_of_vulnerability_statistics_worker_spec.rb
  1. In a group with a mix of projects, set has_vulnerabilities on some: project.project_setting.update!(has_vulnerabilities: true).
  2. Run Feature.enable(:skip_group_archive_jobs_for_projects_without_vulnerabilities, group), then archive the group.
  3. In log/sidekiq.log, both per-project workers ran only for those projects, and the group vulnerability report hides their vulnerabilities; unarchive shows them again.
What was tested
Case Result
Flag on: only projects with the setting on are queued (root group, nested subgroups, project without a settings row, pending-delete project) Pass
Flag off, flag on for another group, flag on for the root group with a subgroup event: all projects queued; flag on for the archived subgroup: filtered Pass
Flag read once per event across many batches, with the archived group as the actor; query count constant as projects grow (warmed-up control) Pass
Empty group, group with projects only in subgroups, deleted group Pass
Negative controls: filter removed, flag forced on, actor from the root namespace, flag read per batch, extra query per project: each fails the spec Pass
GDK, 113 projects: archive and unarchive with the flag on queue exactly the 4 projects with the setting on; report and grades hide and restore them. Flag on for a subgroup only: archiving it filters, archiving the root queues all Pass
GDK: subgroup-only archive, archive then immediate unarchive, 101 subgroups, project archived on its own first Pass
GDK: concurrency limit lowered to 2, 113 jobs deferred and all finished, another caller's job still ran; lock contention unchanged Pass
Specs: 99 examples across the three worker specs, both archived services, the single-project worker and the flag definition; RuboCop clean Pass

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in performance, reliability, security, and maintainability.

Edited by Aanshi Chauhan

Merge request reports

Loading
Loading