Skip to content

Scope batched background migrations to gitlab_schema

Krasimir Angelov requested to merge 359951-scope-bbm-to-gitlab-schema into master

What does this MR do and why?

  • Adds a new column batched_background_migrations.gitlab_schema.
  • Update batched background migration workers to fetch active migration for their connection.
  • Update queue_batched_background_migration helper to set batched_background_migrations.gitlab_schema from allowed_gitlab_schemas when available.

Related to #359951 (closed).

How to set up and validate locally

  • While on master:
  • Stop background jobs - gdk stop rails-background-jobs
  • Simulate schedulling of batched migration:
    gdk psql -d gitlabhq_development -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users"
    gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'users', 'id', 1 FROM users"  
  • Verify the above:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status FROM batched_background_migrations'
  • Change branches - git checkout 359951-scope-bbm-to-gitlab-schema
  • Disable execute_batched_migrations - curl -s https://gitlab.com/-/snippets/2323175/raw/main/patch.diff | git apply
  • Create migration class
    curl -s https://gitlab.com/-/snippets/2323177/raw/main/lib/gitlab/background_migration/test_scoped_bbm.rb > lib/gitlab/background_migration/test_scoped_bbm.rb
  • Migrate - bundle exec rails db:migrate
  • Verify gitlab_schema:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Enable execute_batched_migrations_on_schedule_ci_database feature flag:
    Feature.enable(:execute_batched_migrations_on_schedule_ci_database)
  • Execute CI worker (in Rails console):
    Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
  • Main migration should not be processed for any of the databases:
    gdk psql -d gitlabhq_development -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Execute MAIN worker (in Rails console):
    Database::BatchedBackgroundMigrationWorker.new.perform
  • Records should be migrated for MAIN database:
    gdk psql -d gitlabhq_development -c 'SELECT DISTINCT note FROM users'
  • Records should NOT be migrated for CI database:
    gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT note FROM users'
  • Simulate schedulling of batched migration for CI database:
    gdk psql -d gitlabhq_development_ci -c "INSERT INTO batched_background_migrations (created_at, updated_at, max_value, batch_size, sub_batch_size, interval, job_class_name, table_name, column_name, status, gitlab_schema) SELECT NOW(), NOW(), MAX(id), 1000, 100, 120, 'TestScopedBbm', 'ci_runners', 'id', 1, 'gitlab_ci' FROM ci_runners"`
    gdk psql -d gitlabhq_development_ci -c 'SELECT id, job_class_name, table_name, status, gitlab_schema FROM batched_background_migrations'
  • Execute CI worker (in Rails console):
    Database::BatchedBackgroundMigration::CiDatabaseWorker.new.perform
  • Records should be migrated for CI database:
    gdk psql -d gitlabhq_development_ci -c 'SELECT DISTINCT maintainer_note FROM ci_runners'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Krasimir Angelov

Merge request reports

Loading