Disable beta Orbit enrollments on GitLab.com at GA

Dmitry Gruzdrequested to merge
dgruzd/orbit-beta-enrollment-cleanup into master

What does this MR do and why?

Orbit goes GA and becomes paid on GitLab.com, so groups enabled during the beta must opt in again. An hourly job turns Orbit off for groups still on a beta enrollment, once the access gate is fully on. Groups with the Orbit promo add-on, or opted out of the gate, keep Orbit as direct enrollments. This is the GitLab.com half; the self-managed part comes in a separate MR.

References

Screenshots or screen recordings

No UI changes.

How to set up and validate locally

  1. Run the specs (all pass, including the two-connection concurrency spec; rubocop is clean):

    bundle exec rspec ee/spec/workers/analytics/orbit ee/spec/services/analytics/orbit ee/spec/lib/analytics/orbit/billing/group_account_start_trial_spec.rb ee/spec/services/analytics/knowledge_graph/toggle_namespace_service_spec.rb ee/spec/models/analytics/orbit/namespace_enrollment_spec.rb ee/spec/requests/api/admin/knowledge_graph_spec.rb ee/spec/initializers/1_settings_spec.rb

Performance Analysis

The job runs hourly on GitLab.com and works in batches of 100 beta enrollments, using index scans. See the database review below.

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Rollout checklist

There is no date cutoff. Orbit deletes graph data 30 days after the enabled-namespace delete replicates, so the purge date is the flip time plus 30 days (flip on Oct 30, purge around Nov 29; flip on Oct 31, around Nov 30). The cleanup starts at the first hourly run (:30) after the flip.

  • Precondition (separate MR): orbit_promo_access_gate is type: wip. Move it to gitlab_com_derisk (or beta) with a rollout issue before enabling it in production.
  • Count beta enrollments on a replica (SELECT source, count(*) FROM orbit_namespace_enrollments GROUP BY 1).
  • Opt out GitLab-owned groups first with Feature.opt_out(:orbit_promo_access_gate, group) for gitlab-org (9970) and gitlab-com (6543). Check Feature.opted_out?(:orbit_promo_access_gate, group) for both before the flip.
  • Turn on orbit_trial and orbit_promo_access_gate together on the flip date.
  • Turn the gate fully on (boolean). A percentage-of-actors rollout never starts the cleanup, while a percentage-of-time rollout starts it at random and gates each group at random, so use neither.
  • If the gate is ever turned off globally, feature set orbit_promo_access_gate false also wipes the opt-out actors. Re-apply the opt-outs for 9970 and 6543 before turning it on again.
  • After the flip, check that the hourly job removed the beta rows (disabled_count and converted_count in the job log).

Recovery

The job logs disabled_namespace_ids for every batch it deletes (message Disabled beta Orbit enrollments). To undo a deletion, the group first has to become eligible: opt it out of both orbit_promo_access_gate and orbit_trial, or grant it the orbit_promo add-on. Then re-enable those IDs through the admin API (or ChatOps knowledge_graph_enable_namespace) within 30 days of the delete. That cancels the indexer purge. After the purge the data has to be re-indexed.

Without that, the admin API returns 403 under the gate, since it goes through the same toggle service as the UI. A group opted out of both flags comes back as a beta row, which the next cleanup converts to direct. An add-on group comes back as direct.

For groups that are already enabled, only Feature.opt_out protects gated beta groups. A repeat admin PUT returns early and doesn't convert them anymore.

Database review

Both tables are small. Queries come from DisableBetaEnrollmentsService, which works in batches of 100 beta enrollments. I ran EXPLAIN (ANALYZE, BUFFERS) for each query on a gitlab-production-main thin clone. They all use index scans and each runs in under 15 ms. The full plans are in an internal note, since they show production row counts.

-- batch cursor (each_batch on the primary key, namespace_id)
SELECT namespace_id FROM orbit_namespace_enrollments WHERE source = 1 ORDER BY namespace_id ASC LIMIT 1;
SELECT namespace_id FROM orbit_namespace_enrollments WHERE source = 1 AND namespace_id >= $1 ORDER BY namespace_id ASC LIMIT 1 OFFSET 100;
SELECT namespace_id FROM orbit_namespace_enrollments WHERE source = 1 AND namespace_id >= $1 AND namespace_id < $stop; -- the bound is omitted on the last batch
-- active add-on groups in the batch
SELECT namespace_id FROM subscription_add_on_purchases WHERE subscription_add_on_uid = 10 AND (started_at IS NULL OR started_at <= $today) AND ($today < expires_on) AND namespace_id IN (...);
-- convert (opted-out and add-on groups)
UPDATE orbit_namespace_enrollments SET source = 0, updated_at = now() WHERE source = 1 AND namespace_id IN (...);
-- record_direct: runs after the convert above, from `Billing::Account#record_turned_on` for add-on groups (the toggle service calls it inside its transaction, and the admin PUT goes through that service)
INSERT INTO orbit_namespace_enrollments (namespace_id, source, created_at, updated_at) VALUES ($1, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) ON CONFLICT (namespace_id) DO NOTHING RETURNING namespace_id;
-- in one transaction per batch
SELECT namespace_id FROM orbit_namespace_enrollments WHERE source = 1 AND namespace_id IN (...) FOR UPDATE;
DELETE FROM knowledge_graph_enabled_namespaces WHERE root_namespace_id IN (...);
DELETE FROM orbit_namespace_enrollments WHERE namespace_id IN (...);
Plan shapes (production clone) 
Batch cursor:       Index Scan using orbit_namespace_enrollments_pkey, Filter: source = 1 (plus Index Cond: namespace_id range for the next boundary and the pluck)
Add-on lookup:      Index Scan using index_subscription_add_on_purchases_on_subscription_add_on_uid, Index Cond: subscription_add_on_uid = 10
Convert:            Update -> Index Scan using orbit_namespace_enrollments_pkey, Index Cond: namespace_id = ANY(...), Filter: source = 1
Insert (record_direct): Insert on orbit_namespace_enrollments, Conflict Resolution: NOTHING, Conflict Arbiter Indexes: orbit_namespace_enrollments_pkey
Lock:               LockRows -> Index Scan using orbit_namespace_enrollments_pkey, Index Cond: namespace_id = ANY(...), Filter: source = 1
Delete enabled:     Delete -> Index Scan using index_knowledge_graph_enabled_namespaces_on_root_namespace_id, Index Cond: root_namespace_id = ANY(...)
Delete enrollments: Delete -> Index Scan using orbit_namespace_enrollments_pkey, Index Cond: namespace_id = ANY(...)

Callers: the cleanup service runs the batch cursor, add-on lookup, convert, lock, and both deletes. The toggle service runs only the convert UPDATE and the INSERT, through record_direct (called from Billing::Account#record_turned_on) for add-on groups. The admin PUT reaches it through the same service.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Agent context
  • The worker runs only on GitLab.com and only when the bare flag check is true. The bare check is used because the instance-level check is already true at about 8% of actors on GitLab.com. It defers on database health signals for both tables.
  • The service walks beta enrollments in batches. Groups where the gate is off for the group, and groups with an active add-on, are converted to direct. The rest are deleted in a transaction with the enrollment rows locked, so a concurrent beta-to-trial conversion cannot be clobbered (pinned by a two-connection spec). Enrollments without an enabled row are deleted too.
  • Writers: record_direct (also converts an existing beta row) is called from Billing::Account#record_turned_on for add-on groups, and the admin PUT goes through the toggle service. The toggle service saves the enabled row and the enrollment in one transaction, and record_direct converts beta first, so the cleanup can't delete a fresh opt-in.
  • GroupAccount#after_trial_started records the trial row again after converting beta, so a cleanup before the conversion cannot leave a trial without an enrollment.
  • all_queues.yml generated with gitlab:sidekiq:all_queues_yml:generate; the schedule is in ee/config/schedule_saas.yml (30 * * * *) and the settings spec.
  • Tests: worker, service, model, admin API, toggle service, group account start trial and settings specs.
Edited by Dmitry Gruzd

Merge request reports