Skip to content

Add helper for batched background migrations

Patrick Bair requested to merge pb-add-batched-background-migration-helpers into master

What does this MR do?

In the previous MR !54628 (merged), a new dynamic method of scheduling background migrations that batch over a table was introduced. This MR adds a new migration helper that can be used to kickoff a batched migration, by creating the tracking record in the database.

A previous version of this MR also added a corresponding method that could be used in the down operation to abort a batched migration, but I have removed it for now. I think it's too early to say how we will want to handle this to introduce a method that will most likely change in a future iteration.

For now, we would have to handle the migration rollback manually, which should be fine as this new approach will have a limited rollout in one future migration: !51332 (merged)

The main reason for introducing this MR's queue_batched_background_migration is that it will be used by the backfill_conversion_of_integer_to_bigint method used in the events conversion.

Example migration using them:

class ConvertEventsIdColumnToBigint < ActiveRecord::Migration[6.0]
  include Gitlab::Database::MigrationHelpers

  DOWNTIME = false
  JOB_CLASS_NAME = 'CopyColumnUsingBackgroundMigrationJob'

  def up
    queue_batched_background_migration JOB_CLASS_NAME, :events, :id, 'id', 'id_convert_to_bigint',
      job_interval: 2.minutes, batch_size: 100, sub_batch_size: 25
  end

  def down
    # For now, we would manually abort the appropriate migration records
  end
end

For details about the motivation/plan behind the overarching change, see: #300915 (closed)

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Mayra Cabrera

Merge request reports