Cap enabled DevOps Adoption namespaces per display namespace and stop duplicate snapshot jobs

Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636118

What does this MR do and why?

A Reporter on an Ultimate group can enable DevOps Adoption for many namespaces under that group as the display namespace. Every new row enqueues an Analytics::DevopsAdoption::CreateSnapshotWorker job that scans the namespace's whole subtree on Postgres and Gitaly with no time budget, and nothing stops two jobs for the same namespace from calculating at once. This MR adds the two cheapest controls from the issue behind the devops_adoption_enabled_namespaces_cap feature flag (gitlab_com_derisk, off by default):

  1. A cap of 100 enabled rows per display namespace. EnabledNamespaces::CreateService counts the display namespace's existing rows before saving and returns an error response when the cap is reached. The check lives in CreateService rather than FindOrCreateService because enableDevopsAdoptionNamespace calls CreateService directly and only the bulk path goes through FindOrCreateService; both mutations therefore surface the same error, and finding an existing row is never blocked. The error is also added to the record because both mutations build their errors field from errors_on_object. The count and the insert run under an exclusive lease per display namespace (flag on only), so two concurrent requests near the limit cannot both pass the check. enableDevopsAdoptionNamespace takes a 10 second lease per create; bulkEnableDevopsAdoptionNamespaces takes one 1 minute lease for the whole request and skips the per-row lease, so a 1,000-id request costs two Redis calls, not 2,000. A request that cannot obtain the lease after three retries gets reason: :lock_failed and "Another request is updating this display namespace's enabled namespaces. Try again." in the mutation errors, distinct from the cap error. Duplicate ids in one bulk request are collapsed before the loop. Once a create returns reason: :limit_reached, BulkFindOrCreateService stops creating and only looks up the remaining ids, then returns an error response whose payload still holds every row that exists: rows created before the cap are kept, rows that were already enabled later in the list are still returned, and bulkEnableDevopsAdoptionNamespaces reports the cap once instead of running one count query per remaining id. Any other error response is handled as before. Instance-level rows (no display namespace) are admin-only and stay uncapped.
  2. One snapshot job per namespace at a time. CreateSnapshotWorker takes an exclusive lease keyed on namespace_id with retries: 0 and, when another job holds it, exits with log_extra_metadata_on_done(:result, :skipped). A lease was chosen over deduplicate :until_executed, including_scheduled: true because the job argument is the enabled-namespace row id: rows for the same namespace under different display namespaces have different ids, so Sidekiq deduplication can never collapse them, and rekeying the worker on namespace_id would change both callers and misread any job already queued at deploy time. Skipping is safe because snapshots are keyed by namespace_id, so the job that holds the lease writes the snapshot the skipped job would have written; the nightly CreateAllSnapshotsWorker re-enqueues any row that is still pending. The lease is renewed before each pending range, so the 1 hour TTL only has to outlast the scan of one range, and a lease left behind by a hard-killed job only delays that namespace until the next nightly pass, which is the recovery path the worker already relies on. A single range whose scan takes longer than the TTL would still let a second job in; the issue lists the worker's duration_s p99 as not yet verified, and the TTL is a constant to be sized from that number before the flag is enabled.

With the flag off, both the services and the worker behave exactly as before.

Business impact

A Reporter on an Ultimate group can enable DevOps Adoption for up to 1,000 namespaces per request under that group, and again on the next request. Each enabled row queues a snapshot job that scans the namespace's whole subtree on the main database and calls Gitaly once per project, with no time budget and nothing stopping two jobs for the same namespace from running at once. A Reporter who enables 300 namespaces under one group starts 300 subtree scans on shared infrastructure, and the report page cannot even show them: it lists at most 100 rows per display namespace.

With devops_adoption_enabled_namespaces_cap enabled (off by default), a display namespace holds at most 100 enabled rows and only one snapshot job per namespace runs at a time; a second job exits with result: skipped and the nightly worker re-enqueues anything still pending. A user at the cap sees "Display namespace has reached the limit of 100 enabled namespaces" in the mutation errors; a bulk request that reaches the cap keeps the rows created before it, still returns rows that already existed, and reports the error once. Instance-level rows (admin only) are not capped. This bounds Sidekiq, main PostgreSQL and Gitaly work per display namespace to 100 rows and 100 jobs, and keeps the report consistent with what it can display. With the flag off, nothing changes.

Survivable threshold

100 enabled rows per display namespace preserves everything the product can show today:

  • The report page queries devopsAdoptionEnabledNamespaces with no pagination arguments and never requests further pages, so the schema's default_max_page_size 100 already truncates what a display namespace can display to 100 rows. Rows above 100 are only reachable through the API.
  • The Add or remove subgroups dropdown lists 20 subgroups per search page and enables what the user ticks, so a user working through the UI reaches 100 rows only deliberately.
  • bulkEnableDevopsAdoptionNamespaces accepts up to 1,000 namespace ids per request, and a second request under the same display namespace adds another 1,000 rows and jobs today. With the cap the worst case per display namespace is 100 rows and 100 jobs in total, and further requests add none.

The limit is a constant, CreateService::MAX_ENABLED_NAMESPACES_PER_DISPLAY_NAMESPACE, so it can be raised without a schema change if a customer needs more.

Rollout

  • Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637954
  • The flag is off by default. Enabling it per display namespace (cap) and per namespace (lease) allows a percentage-of-actors rollout.
  • Rollback is turning the flag off: the next create skips the count and the next job skips the lease.

Verification

  • ee/spec/services/analytics/devops_adoption/enabled_namespaces/create_service_spec.rb: cap reached (error, no row, no job), lease obtained around count and insert, a held lease returns the :lock_failed retry error with no row, lock: false skips the lease, two sequential creates one below the limit leave exactly the limit, cap scoped to the display namespace, instance-level rows uncapped, flag off unchanged and lock-free.
  • ee/spec/services/analytics/devops_adoption/enabled_namespaces/bulk_find_or_create_service_spec.rb: one lease for the whole request, a held lease returns the retry error with no rows, a namespace listed twice is created and returned once, rows created before the cap are kept, the cap is reported once, rows already enabled after the cap are still returned, flag off creates the remaining rows without a lease.
  • ee/spec/services/analytics/devops_adoption/enabled_namespaces/find_or_create_service_spec.rb: an existing row is still returned once the cap is reached; #find_enabled_namespace.
  • ee/spec/requests/api/graphql/mutations/analytics/devops_adoption/enabled_namespaces/{enable,bulk_enable}_spec.rb: the cap error is surfaced in the mutation errors field; the bulk mutation returns the rows enabled before and after the cap with the error once.
  • ee/spec/workers/analytics/devops_adoption/create_snapshot_worker_spec.rb: idempotent worker shared example, lease taken and renewed before each range, skip when held, no lease with the flag off.

Local run: all seven files above plus create_all_snapshots_worker_spec.rb in one run, 46 examples, 0 failures; rubocop on the changed files, 10 files inspected, no offenses detected.

Database review

One new query, run once per create while the flag is on. Raw SQL as logged by the spec run (log/test.log):

SELECT COUNT(*) FROM "analytics_devops_adoption_segments" WHERE "analytics_devops_adoption_segments"."display_namespace_id" = <display_namespace_id>

It is served by the existing unique index idx_devops_adoption_segments_namespaces_pair on (display_namespace_id, namespace_id) (db/structure.sql:46683), whose leading column is the one filtered, so the count is an index range scan bounded by the rows of one display namespace; once the cap is on that is at most 100 index entries. No migration. Database Lab query plan: pending; it will be added here when available. The review guidance leaves the ~database label to the author for queries that are not "beyond the obvious" (doc/development/database_review.md, "A database review is required for"), and this single-column equality count on an indexed column is treated as obvious; add the label if a database reviewer is wanted anyway.

Follow-ups

Listed in the issue; left out of this MR so the controls above can ship before the advisory period:

  • One nightly job per distinct namespace_id in CreateAllSnapshotsWorker instead of one per row.
  • Batch the boolean metric queries in SnapshotCalculator the way the counted ones already are.
  • A unique index on analytics_devops_adoption_snapshots (namespace_id, end_time) with first_or_initialize in CalculateAndSaveService, which also closes the issue's separate duplicate-snapshot bug: !267456 (merged).
  • Document the limit in the DevOps adoption user docs when the flag is enabled by default.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chandra Saripaka

Merge request reports

Loading
Loading