Skip to content

Backfill DuoCore for existing subscription in GitlabCom

Qingyu Zhaorequested to merge
527361-backfill-duo-nano-for-existing-subscriptions into master

What does this MR do and why?

For issue #527361+

This MR implements a batched background migration to automatically create DuoCore add-on purchase records for all existing GitLab.com subscriptions with paid plans (including trial plans, but excluding free plans). This ensures that all eligible customers automatically receive access to GitLab Duo Core functionality.

The implementation:

  1. Creates a regular migration to ensure the duo_core add-on exists in the subscription_add_ons table
  2. Implements an EE-only batched background migration that:
    • Identifies eligible GitLab subscriptions (paid plans with valid end dates)
    • Creates corresponding DuoCore add-on purchase records
    • Handles various edge cases (trial subscriptions, null dates, etc.)
  3. Only runs on GitLab.com (excluding JiHu)

Database Changes

This MR includes database changes that:

  • Create a new add-on record in the subscription_add_ons table if it doesn't exist
  • Create new records in the subscription_add_on_purchases table via batched background migration

Database migrate/rollback output

Migrate up:

qingyuzhao@qzhao--20220912-2L0FG gitlab % bundle exec rake db:migrate
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/qingyuzhao/Projects/gdk/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 134260, pg_backend_pid: 46591
main: == 20250423122357 EnsureDuoCoreAddOnExists: migrating =========================
main: == 20250423122357 EnsureDuoCoreAddOnExists: migrated (0.0290s) ================

main: == [advisory_lock_connection] object_id: 134260, pg_backend_pid: 46591
ci: == [advisory_lock_connection] object_id: 134480, pg_backend_pid: 46593
ci: == 20250423122357 EnsureDuoCoreAddOnExists: migrating =========================
ci: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
ci: -- This database can only apply migrations in one of the following schemas: [:gitlab_ci, :gitlab_ci_cell_local, :gitlab_internal, :gitlab_shared].
ci: == 20250423122357 EnsureDuoCoreAddOnExists: migrated (0.0080s) ================

ci: == [advisory_lock_connection] object_id: 134480, pg_backend_pid: 46593
main: == [advisory_lock_connection] object_id: 134700, pg_backend_pid: 46596
main: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: migrating ======
main: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: migrated (0.0394s)

main: == [advisory_lock_connection] object_id: 134700, pg_backend_pid: 46596
ci: == [advisory_lock_connection] object_id: 135080, pg_backend_pid: 46598
ci: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: migrating ======
ci: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
ci: -- This database can only apply migrations in one of the following schemas: [:gitlab_ci, :gitlab_ci_cell_local, :gitlab_internal, :gitlab_shared].
ci: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: migrated (0.0082s)

ci: == [advisory_lock_connection] object_id: 135080, pg_backend_pid: 46598

Migrate down:

qingyuzhao@qzhao--20220912-2L0FG gitlab % bundle exec rake db:migrate:down:main VERSION=20250423131257
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/qingyuzhao/Projects/gdk/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 46971
main: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: reverting ======
main: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: reverted (0.0430s)

main: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 46971
qingyuzhao@qzhao--20220912-2L0FG gitlab %
qingyuzhao@qzhao--20220912-2L0FG gitlab % bundle exec rake db:migrate:down:main VERSION=20250423122357
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/qingyuzhao/Projects/gdk/gitlab/config/environment.rb:7)
main: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47328
main: == 20250423122357 EnsureDuoCoreAddOnExists: reverting =========================
main: == 20250423122357 EnsureDuoCoreAddOnExists: reverted (0.0050s) ================

main: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47328
qingyuzhao@qzhao--20220912-2L0FG gitlab %
qingyuzhao@qzhao--20220912-2L0FG gitlab % bundle exec rake db:migrate:down:ci VERSION=20250423122357
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/qingyuzhao/Projects/gdk/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47532
ci: == 20250423122357 EnsureDuoCoreAddOnExists: reverting =========================
ci: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
ci: -- This database can only apply migrations in one of the following schemas: [:gitlab_ci, :gitlab_ci_cell_local, :gitlab_internal, :gitlab_shared].
ci: == 20250423122357 EnsureDuoCoreAddOnExists: reverted (0.0102s) ================

ci: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47532
qingyuzhao@qzhao--20220912-2L0FG gitlab %
qingyuzhao@qzhao--20220912-2L0FG gitlab %
qingyuzhao@qzhao--20220912-2L0FG gitlab % bundle exec rake db:migrate:down:ci VERSION=20250423131257
DEPRECATION WARNING: Support for Rails versions < 7.1 is deprecated and will be removed from ViewComponent 4.0.0 (ViewComponent v4 will remove support for Rails versions < 7.1 no earlier than April 1, 2025) (called from <main> at /Users/qingyuzhao/Projects/gdk/gitlab/config/environment.rb:7)
ci: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47721
ci: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: reverting ======
ci: -- The migration is skipped since it modifies the schemas: [:gitlab_main].
ci: -- This database can only apply migrations in one of the following schemas: [:gitlab_ci, :gitlab_ci_cell_local, :gitlab_internal, :gitlab_shared].
ci: == 20250423131257 QueueBackfillDuoCoreForExistingSubscription: reverted (0.0099s)

ci: == [advisory_lock_connection] object_id: 133940, pg_backend_pid: 47721
qingyuzhao@qzhao--20220912-2L0FG gitlab %

How to set up and validate locally

  1. Start GDK in Gitlab.com? mode. (in GDK, simulate Gitlab.com using GITLAB_SIMULATE_SAAS=1)
  2. In Gitlab rails console, delete duo_core addon record if it exists
My local test result

There is an existing duo_core AddOn record in my local Gitlab DB, I manually deleted it.

BTW: we could also test the migration script works without deleting it. It is also a valid scenario.

[11] pry(main)> GitlabSubscriptions::AddOn.duo_core
  GitlabSubscriptions::AddOn Load (0.3ms)  SELECT "subscription_add_ons".* FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:<internal:kernel>:187:in `loop'*/
=> [#<GitlabSubscriptions::AddOn:0x0000000318515340
  id: 16,
  created_at: Fri, 25 Apr 2025 00:17:38.552046000 UTC +00:00,
  updated_at: Fri, 25 Apr 2025 00:17:38.552046000 UTC +00:00,
  name: "duo_core",
  description: "[FILTERED]">]
[12] pry(main)>
[13] pry(main)> GitlabSubscriptions::AddOn.duo_core.destroy_all
  GitlabSubscriptions::AddOn Load (0.3ms)  SELECT "subscription_add_ons".* FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):7:in `__pry__'*/
  TRANSACTION (0.1ms)  BEGIN /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):7:in `__pry__'*/
  GitlabSubscriptions::AddOn Destroy (3.2ms)  DELETE FROM "subscription_add_ons" WHERE "subscription_add_ons"."id" = 16 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):7:in `__pry__'*/
  TRANSACTION (0.1ms)  COMMIT /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:/lib/gitlab/database.rb:433:in `commit'*/
=> [#<GitlabSubscriptions::AddOn:0x00000003188b92f8
  id: 16,
  created_at: Fri, 25 Apr 2025 00:17:38.552046000 UTC +00:00,
  updated_at: Fri, 25 Apr 2025 00:17:38.552046000 UTC +00:00,
  name: "duo_core",
  description: "[FILTERED]">]
[14] pry(main)>
[15] pry(main)> GitlabSubscriptions::AddOn.duo_core
  GitlabSubscriptions::AddOn Load (0.2ms)  SELECT "subscription_add_ons".* FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:<internal:kernel>:187:in `loop'*/
=> []
[16] pry(main)>
  1. In Gitlab rails console, prepare some namespaces with the namespace has GitlabSubscription on paid plan, but the namespace does NOT have GitlabSubscriptions::AddOnPurchase.for_duo_core records
My local test result 
[36] pry(main)> GitlabSubscription.with_a_paid_or_trial_hosted_plan.not_expired.count
  GitlabSubscription Count (0.6ms)  SELECT COUNT(*) FROM "gitlab_subscriptions" INNER JOIN "plans" ON "plans"."id" = "gitlab_subscriptions"."hosted_plan_id" WHERE "plans"."name" IN ('bronze', 'silver', 'premium', 'gold', 'ultimate', 'ultimate_trial', 'ultimate_trial_paid_customer', 'premium_trial', 'opensource') AND (end_date IS NULL OR end_date >= '2025-04-26') /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):21:in `__pry__'*/
=> 4
[37] pry(main)>
[38] pry(main)> GitlabSubscription.with_a_paid_or_trial_hosted_plan.not_expired.pluck(:namespace_id)
  GitlabSubscription Pluck (0.4ms)  SELECT "gitlab_subscriptions"."namespace_id" FROM "gitlab_subscriptions" INNER JOIN "plans" ON "plans"."id" = "gitlab_subscriptions"."hosted_plan_id" WHERE "plans"."name" IN ('bronze', 'silver', 'premium', 'gold', 'ultimate', 'ultimate_trial', 'ultimate_trial_paid_customer', 'premium_trial', 'opensource') AND (end_date IS NULL OR end_date >= '2025-04-26') /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):22:in `__pry__'*/
=> [101, 102, 106, 110]
[39] pry(main)>
[40] pry(main)> # Now to check whether they have duo_core records. Make sure some namespaces do NOT have duo_core addonpurchase. The batched background migration script will create for them.
[41] pry(main)>
[42] pry(main)> GitlabSubscriptions::AddOnPurchase.for_duo_core.where(namespace_id: [101, 102, 106, 110]).count
  GitlabSubscriptions::AddOnPurchase Count (0.8ms)  SELECT COUNT(*) FROM "subscription_add_on_purchases" WHERE "subscription_add_on_purchases"."subscription_add_on_id" IN (SELECT "subscription_add_ons"."id" FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5) AND "subscription_add_on_purchases"."namespace_id" IN (101, 102, 106, 110) /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):24:in `__pry__'*/
=> 0
[43] pry(main)>

  1. Run the migrations:

    bundle exec rake db:migrate

  2. Verify the duo_core add-on exists in the subscription_add_ons table. Verify that duo_core add-on-purchases are created for the eligible namespaces

My local test result 
[45] pry(main)> GitlabSubscriptions::AddOn.duo_core
  GitlabSubscriptions::AddOn Load (0.7ms)  SELECT "subscription_add_ons".* FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:<internal:kernel>:187:in `loop'*/
=> [#<GitlabSubscriptions::AddOn:0x0000000317273660
  id: 17,
  created_at: Sat, 26 Apr 2025 01:24:10.974747000 UTC +00:00,
  updated_at: Sat, 26 Apr 2025 01:24:10.974747000 UTC +00:00,
  name: "duo_core",
  description: "[FILTERED]">]
[46] pry(main)>
[47] pry(main)> GitlabSubscriptions::AddOnPurchase.for_duo_core.where(namespace_id: [101, 102, 106, 110]).count
  GitlabSubscriptions::AddOnPurchase Count (0.5ms)  SELECT COUNT(*) FROM "subscription_add_on_purchases" WHERE "subscription_add_on_purchases"."subscription_add_on_id" IN (SELECT "subscription_add_ons"."id" FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5) AND "subscription_add_on_purchases"."namespace_id" IN (101, 102, 106, 110) /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):27:in `__pry__'*/
=> 3
[48] pry(main)> GitlabSubscriptions::AddOnPurchase.for_duo_core.where(namespace_id: [101, 102, 106, 110])
  GitlabSubscriptions::AddOnPurchase Load (0.4ms)  SELECT "subscription_add_on_purchases".* FROM "subscription_add_on_purchases" WHERE "subscription_add_on_purchases"."subscription_add_on_id" IN (SELECT "subscription_add_ons"."id" FROM "subscription_add_ons" WHERE "subscription_add_ons"."name" = 5) AND "subscription_add_on_purchases"."namespace_id" IN (101, 102, 106, 110) /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:<internal:kernel>:187:in `loop'*/
=> [#<GitlabSubscriptions::AddOnPurchase:0x00000003115588e0
  id: 75,
  created_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  updated_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  subscription_add_on_id: 17,
  namespace_id: 101,
  quantity: 123456,
  expires_on: Sat, 18 Apr 2026,
  purchase_xid: "duo_core_backfill_2025",
  last_assigned_users_refreshed_at: nil,
  trial: false,
  started_at: Fri, 04 Apr 2025,
  organization_id: 1>,
 #<GitlabSubscriptions::AddOnPurchase:0x0000000311558660
  id: 76,
  created_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  updated_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  subscription_add_on_id: 17,
  namespace_id: 106,
  quantity: 10000,
  expires_on: Sat, 21 Jun 2025,
  purchase_xid: "duo_core_backfill_2025",
  last_assigned_users_refreshed_at: nil,
  trial: false,
  started_at: Tue, 22 Apr 2025,
  organization_id: 1>,
 #<GitlabSubscriptions::AddOnPurchase:0x0000000311557e40
  id: 77,
  created_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  updated_at: Sat, 26 Apr 2025 01:24:17.178395000 UTC +00:00,
  subscription_add_on_id: 17,
  namespace_id: 110,
  quantity: 10000,
  expires_on: Sat, 21 Jun 2025,
  purchase_xid: "duo_core_backfill_2025",
  last_assigned_users_refreshed_at: nil,
  trial: false,
  started_at: Tue, 22 Apr 2025,
  organization_id: 1>]
[49] pry(main)>

NOTE: the namespace 102 did NOT get duo_core add-on-purchase created, due to the namespace 102 is NOT a root namespace. It has parent_id. We only create duo_core add-on-purchase for root namespace. This is expected behaviour.

[49] pry(main)> Namespace.find(102).parent_id
  Namespace Load (3.2ms)  SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM find_namespaces_by_id(102) AS namespaces WHERE ("namespaces"."id" IS NOT NULL) LIMIT 1 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:qzhao--20220912-2L0FG,console_username:qingyuzhao,line:(pry):29:in `__pry__'*/
=> 101

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #527361

Edited by Qingyu Zhao

Merge request reports