Split diff commit migrations into smaller chunks
What does this MR do?
On GitLab.com we are running into issues where batch sizes for the migration MigrateMergeRequestDiffCommitUsers are too large, resulting in Sidekiq hosts running out of memory. See issue https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14097 for more details on this.
To work around this, we add a migration that slices the existing jobs into smaller chunks. This done such that jobs using the old ranges won't be processed any more. We also change the number of database rows loaded into memory when migrating data, in an attempt to further reduce the amount of memory that's necessary.
See #334394 (closed) for more information.
Database
Time estimate for the background migration: ~2 weeks
Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Edited by Andy Schoenen