Skip to content

Scope batched background migrations to gitlab_schema

Krasimir Angelov requested to merge 359951-scope-bbm-to-gitlab-schema into master

What does this MR do and why?

  • Adds a new column batched_background_migrations.gitlab_schema.
  • Update batched background migration workers to fetch active migration for their connection.
  • Update queue_batched_background_migration helper to set batched_background_migrations.gitlab_schema from allowed_gitlab_schemas when available.

Related to #359951 (closed).

How to set up and validate locally

  • While on master:
  • Stop background jobs - gdk stop rails-background-jobs
  • Simulate schedulling of batched migration:
    gdk psql -d gitlabhq_development -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users"
    gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users"  
  • Verify the above:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations'
  • Change branches - git checkout 359951-scope-bbm-to-gitlab-schema
  • Disable execute_batched_migrations - curl -s https://gitlab.com/-/snippets/2323175/raw/main/patch.diff | git apply
  • Create migration class
    curl -s https://gitlab.com/-/snippets/2323177/raw/main/lib/gitlab/background_migration/test_scoped_bbm.rb > lib/gitlab/background_migration/test_scoped_bbm.rb
  • Migrate - bundle exec rails db:migrate
  • Verify gitlab_schema:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Enable execute_batched_migrations_on_schedule_ci_database feature flag:
    Feature.enable(:execute_batched_migrations_on_schedule_ci_database)
  • Execute CI worker (in Rails console):
    Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
  • Main migration should not be processed for any of the databases:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Execute MAIN worker (in Rails console):
    Database::BatchedBackgroundMigrationWorker.new.perform
  • Records should be migrated for MAIN database:
    gdk psql -d gitlabhq_development -c 'SELECT DISTINCT note FROM users'
  • Records should NOT be migrated for CI database:
    gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT note FROM users'
  • Simulate schedulling of batched migration for CI database:
    gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status, gitlab_schema) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'ci_runners', 'id', 1, 'gitlab_ci' FROM ci_runners"`
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Execute CI worker (in Rails console):
    Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
  • Records should be migrated for CI database:
    gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT maintainer_note FROM ci_runners'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Krasimir Angelov

Merge request reports