Cap enabled DevOps Adoption namespaces per display namespace and stop duplicate snapshot jobs
Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636118
What does this MR do and why?
A Reporter on an Ultimate group can enable DevOps Adoption for many namespaces under that group as the display namespace. Every new row enqueues an Analytics::DevopsAdoption::CreateSnapshotWorker job that scans the namespace's whole subtree on Postgres and Gitaly with no time budget, and nothing stops two jobs for the same namespace from calculating at once. This MR adds the two cheapest controls from the issue behind the devops_adoption_enabled_namespaces_cap feature flag (gitlab_com_derisk, off by default):
- A cap of 100 enabled rows per display namespace.
EnabledNamespaces::CreateServicecounts the display namespace's existing rows before saving and returns an error response when the cap is reached. The check lives inCreateServicerather thanFindOrCreateServicebecauseenableDevopsAdoptionNamespacecallsCreateServicedirectly and only the bulk path goes throughFindOrCreateService; both mutations therefore surface the same error, and finding an existing row is never blocked. The error is also added to the record because both mutations build theirerrorsfield fromerrors_on_object. The count and the insert run under an exclusive lease per display namespace (flag on only), so two concurrent requests near the limit cannot both pass the check.enableDevopsAdoptionNamespacetakes a 10 second lease per create;bulkEnableDevopsAdoptionNamespacestakes one 1 minute lease for the whole request and skips the per-row lease, so a 1,000-id request costs two Redis calls, not 2,000. A request that cannot obtain the lease after three retries getsreason: :lock_failedand "Another request is updating this display namespace's enabled namespaces. Try again." in the mutationerrors, distinct from the cap error. Duplicate ids in one bulk request are collapsed before the loop. Once a create returnsreason: :limit_reached,BulkFindOrCreateServicestops creating and only looks up the remaining ids, then returns an error response whose payload still holds every row that exists: rows created before the cap are kept, rows that were already enabled later in the list are still returned, andbulkEnableDevopsAdoptionNamespacesreports the cap once instead of running one count query per remaining id. Any other error response is handled as before. Instance-level rows (no display namespace) are admin-only and stay uncapped. - One snapshot job per namespace at a time.
CreateSnapshotWorkertakes an exclusive lease keyed onnamespace_idwithretries: 0and, when another job holds it, exits withlog_extra_metadata_on_done(:result, :skipped). A lease was chosen overdeduplicate :until_executed, including_scheduled: truebecause the job argument is the enabled-namespace row id: rows for the same namespace under different display namespaces have different ids, so Sidekiq deduplication can never collapse them, and rekeying the worker onnamespace_idwould change both callers and misread any job already queued at deploy time. Skipping is safe because snapshots are keyed bynamespace_id, so the job that holds the lease writes the snapshot the skipped job would have written; the nightlyCreateAllSnapshotsWorkerre-enqueues any row that is still pending. The lease is renewed before each pending range, so the 1 hour TTL only has to outlast the scan of one range, and a lease left behind by a hard-killed job only delays that namespace until the next nightly pass, which is the recovery path the worker already relies on. A single range whose scan takes longer than the TTL would still let a second job in; the issue lists the worker'sduration_sp99 as not yet verified, and the TTL is a constant to be sized from that number before the flag is enabled.
With the flag off, both the services and the worker behave exactly as before.
Business impact
A Reporter on an Ultimate group can enable DevOps Adoption for up to 1,000 namespaces per request under that group, and again on the next request. Each enabled row queues a snapshot job that scans the namespace's whole subtree on the main database and calls Gitaly once per project, with no time budget and nothing stopping two jobs for the same namespace from running at once. A Reporter who enables 300 namespaces under one group starts 300 subtree scans on shared infrastructure, and the report page cannot even show them: it lists at most 100 rows per display namespace.
With devops_adoption_enabled_namespaces_cap enabled (off by default), a display namespace holds at most 100 enabled rows and only one snapshot job per namespace runs at a time; a second job exits with result: skipped and the nightly worker re-enqueues anything still pending. A user at the cap sees "Display namespace has reached the limit of 100 enabled namespaces" in the mutation errors; a bulk request that reaches the cap keeps the rows created before it, still returns rows that already existed, and reports the error once. Instance-level rows (admin only) are not capped. This bounds Sidekiq, main PostgreSQL and Gitaly work per display namespace to 100 rows and 100 jobs, and keeps the report consistent with what it can display. With the flag off, nothing changes.
Survivable threshold
100 enabled rows per display namespace preserves everything the product can show today:
- The report page queries
devopsAdoptionEnabledNamespaceswith no pagination arguments and never requests further pages, so the schema'sdefault_max_page_size 100already truncates what a display namespace can display to 100 rows. Rows above 100 are only reachable through the API. - The Add or remove subgroups dropdown lists 20 subgroups per search page and enables what the user ticks, so a user working through the UI reaches 100 rows only deliberately.
bulkEnableDevopsAdoptionNamespacesaccepts up to 1,000 namespace ids per request, and a second request under the same display namespace adds another 1,000 rows and jobs today. With the cap the worst case per display namespace is 100 rows and 100 jobs in total, and further requests add none.
The limit is a constant, CreateService::MAX_ENABLED_NAMESPACES_PER_DISPLAY_NAMESPACE, so it can be raised without a schema change if a customer needs more.
Rollout
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637954
- The flag is off by default. Enabling it per display namespace (cap) and per namespace (lease) allows a percentage-of-actors rollout.
- Rollback is turning the flag off: the next create skips the count and the next job skips the lease.
Verification
ee/spec/services/analytics/devops_adoption/enabled_namespaces/create_service_spec.rb: cap reached (error, no row, no job), lease obtained around count and insert, a held lease returns the:lock_failedretry error with no row,lock: falseskips the lease, two sequential creates one below the limit leave exactly the limit, cap scoped to the display namespace, instance-level rows uncapped, flag off unchanged and lock-free.ee/spec/services/analytics/devops_adoption/enabled_namespaces/bulk_find_or_create_service_spec.rb: one lease for the whole request, a held lease returns the retry error with no rows, a namespace listed twice is created and returned once, rows created before the cap are kept, the cap is reported once, rows already enabled after the cap are still returned, flag off creates the remaining rows without a lease.ee/spec/services/analytics/devops_adoption/enabled_namespaces/find_or_create_service_spec.rb: an existing row is still returned once the cap is reached;#find_enabled_namespace.ee/spec/requests/api/graphql/mutations/analytics/devops_adoption/enabled_namespaces/{enable,bulk_enable}_spec.rb: the cap error is surfaced in the mutationerrorsfield; the bulk mutation returns the rows enabled before and after the cap with the error once.ee/spec/workers/analytics/devops_adoption/create_snapshot_worker_spec.rb: idempotent worker shared example, lease taken and renewed before each range, skip when held, no lease with the flag off.
Local run: all seven files above plus create_all_snapshots_worker_spec.rb in one run, 46 examples, 0 failures; rubocop on the changed files, 10 files inspected, no offenses detected.
Database review
One new query, run once per create while the flag is on. Raw SQL as logged by the spec run (log/test.log):
SELECT COUNT(*) FROM "analytics_devops_adoption_segments" WHERE "analytics_devops_adoption_segments"."display_namespace_id" = <display_namespace_id>It is served by the existing unique index idx_devops_adoption_segments_namespaces_pair on (display_namespace_id, namespace_id) (db/structure.sql:46683), whose leading column is the one filtered, so the count is an index range scan bounded by the rows of one display namespace; once the cap is on that is at most 100 index entries. No migration. Database Lab query plan: pending; it will be added here when available. The review guidance leaves the ~database label to the author for queries that are not "beyond the obvious" (doc/development/database_review.md, "A database review is required for"), and this single-column equality count on an indexed column is treated as obvious; add the label if a database reviewer is wanted anyway.
Follow-ups
Listed in the issue; left out of this MR so the controls above can ship before the advisory period:
- One nightly job per distinct
namespace_idinCreateAllSnapshotsWorkerinstead of one per row. - Batch the boolean metric queries in
SnapshotCalculatorthe way the counted ones already are. - A unique index on
analytics_devops_adoption_snapshots (namespace_id, end_time)withfirst_or_initializeinCalculateAndSaveService, which also closes the issue's separate duplicate-snapshot bug: !267456 (merged). - Document the limit in the DevOps adoption user docs when the flag is enabled by default.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.