Draft: Improve scheduling of diff commit background jobs
What does this MR do?
This improves how MigrateMergeRequestDiffCommitUsers jobs are scheduled. Prior to this commit, we would schedule jobs at a fixed (and increasing) interval. This worked for about 50% of the work on GitLab.com, but the other 50% started running into issues. In particular, if one job takes a very long time (e.g. 30 minutes), other jobs will be rescheduled/retried during that time. If that happens often enough, the jobs are discarded.
In this commit, the logic is changed such that each job schedules the next pending job. In addition, if a job has already been processed, we bail out. This ensures that jobs still sitting in the Sidekiq scheduled set aren't processed multiple times.
In this commit we also include a migration that kicks off this process by scheduling a single pending job (if any). This removes the need for manual scheduling in production.
See the thread in #334394 (comment 625077286) for more information.
Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) -
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) -
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.