Skip to content

Spread sidekiq jobs on reschedule during github import

What does this MR do and why?

In Github importer we are rescheduling all scheduled jobs that fail to import when the rate limit has been reached that causing to schedule all of that jobs to be enqueued at the same time.

With this MR the same delay calculation is used as during jobs spreading upon the first schedule of jobs to have it in batches, so when the rate limit is reset it just double the number of scheduled jobs to be enqueued at the same time (one batch of reschedule and one batch of first schedule)

Screenshots or screen recordings

Scheduled pull request jobs:

Before

Screenshot_2023-04-14_at_23.14.59

After

Screenshot_2023-04-14_at_23.15.20

How to set up and validate locally

It's a bit time-consuming to reproduce the issue.

  1. Initiate an import of a large project to reach ratel limit during import
  2. Observe Sidekiq logs for scheduled_at, there shouldn't be a lot of jobs scheduled at the same time.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Kristina Doskich

Merge request reports