Support multiple databases for batched background migrations

Batched background migrations need to be updated to support multiple databases, or we will not be able to use them for new migrations leading up to or after the release of database decomposition. Existing migrations that have completed on GitLab.com will be fine, since there are no plans to rollout decomposition to self-managed customers.

We are already updating the redis-based background migrations to work with multiple databases, so this issue is not required to support database decomposition. We will have to support multiple databases before we can consider batched migrations ready for general use, so it should be done in an early iteration of future work.

As a high level to implement this, we have previously discussed:

  • Convert batched migration models to be SharedModel
  • Add a sidekiq cron specifically for the ci database, which is a separate processing queue from main
  • Update rake tasks to execute migrations to accept a database argument
  • Wrap all entry points to migration execution with SharedModel.using_connection with the proper connection
  • Update generic migration job CopyColumnUsingBackgroundMigrationJob to use the correct connection
  • Update generic batching strategy PrimaryKeyBatchingStrategy to use the correct connection
Edited Mar 15, 2022 by Diogo Frazão
Assignee Loading
Time tracking Loading