Skip to content

Add migration_plan to container_repositories

Steve Abrams requested to merge phase2-add-plan-to-container-repositories into master

🔎 What does this MR do and why?

We are moving all container repositories to the new registry on GitLab.com. In Phase 2 of this move, we are importing existing container repositories in several stages by customer tier (or plan).

Container repositories are far removed from their respective plan: container_repositories -> projects -> namespaces -> gitlab_subscriptions -> plans.

We need the ability to quickly query the number of repositories in each plan and see their current migration state. Joining against this many tables is not performant, so we are going to add a migration_plan column to container_repositories to hold the plan.name value for easy access.

We know this value may become inaccurate over time as different groups/namespaces change tiers, but that is ok, we may refresh these values along the way, and if a small number of container repositories get imported with a certain plan that they no longer belong to, that is an acceptable outcome.

🐘 Database

Background migration

See the database testing output for the EachBatch queries run in the post-deployment migration. Since we are doing a full table iteration with no conditions on container_repositories, there are no surprises and timings look good.

Updating a container_repository record. Here are a few sample queries, averaging around ~50ms each:

Each job:

Batch size: 1500
Update timing: 50ms


(1500 * 50ms) / 1000 = 75 sec per job 
This gives some room to spare in the 120 sec max job time target

Total migration:

Rows to be updated: 2112613
Batch size: 1500
Total batches/jobs: 2112613 / 1500 = 1409
Job Delay: 2 minutes
Total Run Time: (1409 * 2 min) / 60 min per hour / 24 hours per day = 1.96 Days

Migrations

Up:

== 20220316202200 AddMigrationPlanToContainerRepositories: migrating ==========
-- add_column(:container_repositories, :migration_plan, :text)
   -> 0.0030s
== 20220316202200 AddMigrationPlanToContainerRepositories: migrated (0.0030s) =

== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: migrating =
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE container_repositories\nADD CONSTRAINT check_05e9012f36\nCHECK ( char_length(migration_plan) <= 255 )\nNOT VALID;\n")
   -> 0.0014s
-- current_schema()
   -> 0.0002s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE container_repositories VALIDATE CONSTRAINT check_05e9012f36;")
   -> 0.0010s
-- execute("RESET statement_timeout")
   -> 0.0005s
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: migrated (0.0216s)

== 20220316202640 PopulateContainerRepositoriesMigrationPlan: migrating =======
-- Scheduled 0 PopulateContainerRepositoryMigrationPlan jobs with a maximum of 1500 records per batch and an interval of 120 seconds.

The migration is expected to take at least 0 seconds. Expect all jobs to have completed after 2022-03-17 01:44:01 UTC."
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: migrated (0.0216s)

Down:

== 20220316202640 PopulateContainerRepositoriesMigrationPlan: reverting =======
== 20220316202640 PopulateContainerRepositoriesMigrationPlan: reverted (0.0000s)

== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: reverting =
-- transaction_open?()
   -> 0.0000s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE container_repositories\nDROP CONSTRAINT IF EXISTS check_05e9012f36\n")
   -> 0.0013s
== 20220316202402 AddTextLimitToContainerRepositoriesMigrationPlan: reverted (0.0167s)

== 20220316202200 AddMigrationPlanToContainerRepositories: reverting ==========
-- remove_column(:container_repositories, :migration_plan, :text)
   -> 0.0026s
== 20220316202200 AddMigrationPlanToContainerRepositories: reverted (0.0098s) =

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Create some container repositories locally if you do not already have some 
    10.times { FactoryBot.create(:container_repository, project: Project.first) }
  2. Change the project's tier 
    Plan.create!(name: 'ultimate')
GitlabSubscription.create(hosted_plan_id: Plan.last.id, namespace_id: Project.first.namespace.id)
  3. Run the migrations locally
  4. Ensure the background migration sidekiq jobs have run locally (only one job should have been created unless you have over 1500 local container repositories)
  5. The container_repositories should now show the new plan: 
    ContainerRepository.first.migration_plan
=> 'ultimate'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #356218 (closed)

Edited by Steve Abrams

Merge request reports