Scope batched background migrations to gitlab_schema
What does this MR do and why?
- Adds a new column
batched_background_migrations.gitlab_schema
. - Update batched background migration workers to fetch active migration for their connection.
- Update
queue_batched_background_migration
helper to setbatched_background_migrations.gitlab_schema
fromallowed_gitlab_schemas
when available.
Related to #359951 (closed).
How to set up and validate locally
- While on master:
- Stop background jobs -
gdk stop rails-background-jobs
- Simulate schedulling of batched migration:
gdk psql -d gitlabhq_development -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users" gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users"
- Verify the above:
gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations' gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations'
- Change branches -
git checkout 359951-scope-bbm-to-gitlab-schema
- Disable
execute_batched_migrations
-curl -s https://gitlab.com/-/snippets/2323175/raw/main/patch.diff | git apply
- Create migration class
curl -s https://gitlab.com/-/snippets/2323177/raw/main/lib/gitlab/background_migration/test_scoped_bbm.rb > lib/gitlab/background_migration/test_scoped_bbm.rb
- Migrate -
bundle exec rails db:migrate
- Verify
gitlab_schema
:gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations' gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
- Enable execute_batched_migrations_on_schedule_ci_database feature flag:
Feature.enable(:execute_batched_migrations_on_schedule_ci_database)
- Execute CI worker (in Rails console):
Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
- Main migration should not be processed for any of the databases:
gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations' gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
- Execute MAIN worker (in Rails console):
Database::BatchedBackgroundMigrationWorker.new.perform
- Records should be migrated for MAIN database:
gdk psql -d gitlabhq_development -c 'SELECT DISTINCT note FROM users'
- Records should NOT be migrated for CI database:
gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT note FROM users'
- Simulate schedulling of batched migration for CI database:
gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status, gitlab_schema) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'ci_runners', 'id', 1, 'gitlab_ci' FROM ci_runners"` gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
- Execute CI worker (in Rails console):
Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
- Records should be migrated for CI database:
gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT maintainer_note FROM ci_runners'
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Krasimir Angelov