Skip to content

Split diff commit migrations into smaller chunks

What does this MR do?

On GitLab.com we are running into issues where batch sizes for the migration MigrateMergeRequestDiffCommitUsers are too large, resulting in Sidekiq hosts running out of memory. See issue https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14097 for more details on this.

To work around this, we add a migration that slices the existing jobs into smaller chunks. This done such that jobs using the old ranges won't be processed any more. We also change the number of database rows loaded into memory when migrating data, in an attempt to further reduce the amount of memory that's necessary.

See #334394 (closed) for more information.

Database

Time estimate for the background migration: ~2 weeks

Does this MR meet the acceptance criteria?

Conformity

Edited by Andy Schoenen

Merge request reports