Skip to content

Draft: Improve scheduling of diff commit background jobs

What does this MR do?

This improves how MigrateMergeRequestDiffCommitUsers jobs are scheduled. Prior to this commit, we would schedule jobs at a fixed (and increasing) interval. This worked for about 50% of the work on GitLab.com, but the other 50% started running into issues. In particular, if one job takes a very long time (e.g. 30 minutes), other jobs will be rescheduled/retried during that time. If that happens often enough, the jobs are discarded.

In this commit, the logic is changed such that each job schedules the next pending job. In addition, if a job has already been processed, we bail out. This ensures that jobs still sitting in the Sidekiq scheduled set aren't processed multiple times.

In this commit we also include a migration that kicks off this process by scheduling a single pending job (if any). This removes the need for manual scheduling in production.

See the thread in #334394 (comment 625077286) for more information.

Does this MR meet the acceptance criteria?

Conformity

Merge request reports