Skip to content

Select worker to schedule BG migrations

Patrick Bair requested to merge 343055-put-bg-migrations-in-correct-queue into master

What does this MR do and why?

Related to #343055 (closed)

Prepares for the addition of a separate BG migration worker for the ci database by conditionally scheduling jobs based on the target database of the migration. The target database for a BG migration is the "primary" database for the migration, where tracking data for that migration will live.

While a migration job is unrestricted to use connections to either database, a developer adding a BG migration will have to choose the database/queue where the migration information will be tracked and executed. For most BG migrations that only target tables in a single database, the BG migration will be queued to also use the tracking tables in the same database.

For now we only support scheduling to the main database, but we will add the working and scheduling for the ci database in a future change.

How to set up and validate locally

  1. Create a test background migration in the rails console:
    module Gitlab
      module BackgroundMigration
        class MyFancyMigration
          def perform(start_id, end_id)
            puts "fancy: #{start_id}, #{end_id}"
          end
        end
      end
    end
  2. Create and run a migration to enqueue the jobs for a specific table:
    migration = Class.new(Gitlab::Database::Migration[1.0]) do
      def up
        queue_background_migration_jobs_by_range_at_intervals(define_batchable_model(:issues), 'MyFancyMigration', 2.minutes, batch_size: 2, initial_delay: 1.week, track_jobs: true)
      end
    end
    
    migration.new.up
    # -- Scheduled 39 MyFancyMigration jobs with a maximum of 2 records per batch and an interval of 120 seconds.
  3. Verify the scheduled set matches the number of jobs reported above:
    Sidekiq::ScheduledSet.new.select { |j| j.klass == 'BackgroundMigrationWorker' && j.args.first == 'MyFancyMigration' }.size
    # 39
  4. Create and run a migration to finalize and force the jobs to run:
    migration2 = Class.new(Gitlab::Database::Migration[1.0]) do
      def up
        finalize_background_migration('MyFancyMigration')
      end
    end
    
    migration2.new.up
    # you should see the job output as they run
  5. Verify the queue is empty, and the migrations are complete:
    Sidekiq::ScheduledSet.new.select { |j| j.klass == 'BackgroundMigrationWorker' && j.args.first == 'MyFancyMigration' }.size
    # 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Patrick Bair

Merge request reports