Queue group archive vulnerability jobs only for vulnerable projects
What does this MR do and why?
Archiving or unarchiving a group queues two jobs for every project in it, even projects with no vulnerabilities. Those jobs find nothing, but each takes a Sidekiq slot. On GitLab.com, 8,410 jobs came from this in 5 days. A 1,248-project subgroup queued 2,496 jobs where 0 were needed. A 736-project root group queued 1,472 where 460 were needed.
- Behind the new
skip_group_archive_jobs_for_projects_without_vulnerabilitiesfeature flag (off by default), jobs are queued only for projects whosehas_vulnerabilitiessetting is on. The flag is read once per event, with the archived group as the actor. - Both per-project workers get a concurrency limit of 25 and defer on the database health signal. This is not behind the flag.
- The worker walks groups only, skipping project namespaces.
Review notes
Alternatives considered: Filtering inside each per-project job instead of before queueing (rejected: the job count, the actual cost, would not drop). Putting the concurrency limit behind the flag too (not done: other callers of the reads worker queued 16 jobs in 5 days against 8,410 from archives, so the limit barely touches them).
Not done, tracked at: Batching the per-project jobs: second MR for the same issue https://gitlab.com/gitlab-org/gitlab/-/work_items/637720 (needs its own feature flag because the workers take one id today). The 5-minute per-project lock is shorter than the longest jobs (13 minutes), so two jobs for one project can overlap: https://gitlab.com/gitlab-org/gitlab/-/work_items/638907. Ingestion during a group archive writes archived = false because it uses the project's own archived column: pre-existing, no issue yet. Per-project route lookup when building job contexts: pre-existing, no issue yet.
- Approvals: backend maintainer (CODEOWNERS
[Maintainers]coversee/andee/config/feature_flags/). Nodb/files change; the Database review section below covers the one changed query. - Deviation from the issue: the issue did not ask for
namespace_class: Groupor the database health deferral; both mirror the merged sibling workerSecurity::AnalyzersStatus::ProcessGroupArchivedEventsWorker/UpdateArchivedAnalyzerStatusWorker. - Why 25: the value the sibling security workers use; locally, with the limit at 2, 111 of 113 jobs per worker were deferred and all finished, so deferred jobs are never dropped. The largest production archive ran 6-7 jobs at once, under 25.
- Flag actor: the archived group, so the flag can be enabled for one test group inside
gitlab-org. A subgroup archived on its own without the flag queues every project, which is today's behaviour. - Known risk, accepted: a project with vulnerabilities but
has_vulnerabilitiesoff would be skipped and keep a stalearchivedvalue. A production check of both groups found none. The rollout issue holds the check and the repair step: https://gitlab.com/gitlab-org/gitlab/-/work_items/638850 - Accepted side effect: leftover
vulnerability_statisticsrows with zero counts and the setting off keep a stalearchivedvalue; the grades panel already excludes archived projects throughProject.non_archived.
Database review
The only changed statement is the per-batch project load, which now joins project_settings and uses the partial index index_project_settings_on_project_id_partially.
SQL, plans and reading
One real batch on gitlab-org (9970): the 100 lowest-id projects in its tree, which sit in 5 namespaces.
Before:
SELECT "projects".* FROM "projects" WHERE "projects"."namespace_id" IN (9970, 1793858, 1920510, 2287432, 14018650) AND "projects"."id" >= 13083 AND "projects"."id" < 3605986After:
SELECT "projects".* FROM "projects" INNER JOIN "project_settings" ON "project_settings"."project_id" = "projects"."id" WHERE "projects"."namespace_id" IN (9970, 1793858, 1920510, 2287432, 14018650) AND "projects"."id" >= 13083 AND "projects"."id" < 3605986 AND (has_vulnerabilities IS TRUE)| Statement | Plan | Rows | Buffers (warm) | Execution (warm) |
|---|---|---|---|---|
| Before | https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82707/commands/190227 | 100 | 120 | 0.4 ms |
| After | https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82707/commands/190229 | 30 | 425 | 0.6 ms |
projectsusesindex_projects_on_namespace_id_and_idin both plans (120 buffers).project_settingsis an index-only scan onindex_project_settings_on_project_id_partially, once per project in the batch (305 buffers). That is the whole difference.
How to set up and validate locally
bin/rspec ee/spec/workers/vulnerabilities/process_group_archived_events_worker_spec.rb ee/spec/workers/vulnerabilities/update_archived_attribute_of_vulnerability_reads_worker_spec.rb ee/spec/workers/vulnerabilities/update_archived_attribute_of_vulnerability_statistics_worker_spec.rb- In a group with a mix of projects, set
has_vulnerabilitieson some:project.project_setting.update!(has_vulnerabilities: true). - Run
Feature.enable(:skip_group_archive_jobs_for_projects_without_vulnerabilities, group), then archive the group. - In
log/sidekiq.log, both per-project workers ran only for those projects, and the group vulnerability report hides their vulnerabilities; unarchive shows them again.
What was tested
| Case | Result |
|---|---|
| Flag on: only projects with the setting on are queued (root group, nested subgroups, project without a settings row, pending-delete project) | Pass |
| Flag off, flag on for another group, flag on for the root group with a subgroup event: all projects queued; flag on for the archived subgroup: filtered | Pass |
| Flag read once per event across many batches, with the archived group as the actor; query count constant as projects grow (warmed-up control) | Pass |
| Empty group, group with projects only in subgroups, deleted group | Pass |
| Negative controls: filter removed, flag forced on, actor from the root namespace, flag read per batch, extra query per project: each fails the spec | Pass |
| GDK, 113 projects: archive and unarchive with the flag on queue exactly the 4 projects with the setting on; report and grades hide and restore them. Flag on for a subgroup only: archiving it filters, archiving the root queues all | Pass |
| GDK: subgroup-only archive, archive then immediate unarchive, 101 subgroups, project archived on its own first | Pass |
| GDK: concurrency limit lowered to 2, 113 jobs deferred and all finished, another caller's job still ran; lock contention unchanged | Pass |
| Specs: 99 examples across the three worker specs, both archived services, the single-project worker and the flag definition; RuboCop clean | Pass |
References
- Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/637720
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638850
- Parent: https://gitlab.com/gitlab-org/gitlab/-/work_items/636205
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in performance, reliability, security, and maintainability.