Spread sidekiq jobs on reschedule during github import
What does this MR do and why?
In Github importer we are rescheduling all scheduled jobs that fail to import when the rate limit has been reached that causing to schedule all of that jobs to be enqueued at the same time.
With this MR the same delay calculation is used as during jobs spreading upon the first schedule of jobs to have it in batches, so when the rate limit is reset it just double the number of scheduled jobs to be enqueued at the same time (one batch of reschedule and one batch of first schedule)
Screenshots or screen recordings
Scheduled pull request jobs:
Before
After
How to set up and validate locally
It's a bit time-consuming to reproduce the issue.
- Initiate an import of a large project to reach ratel limit during import
- Observe Sidekiq logs for
scheduled_at
, there shouldn't be a lot of jobs scheduled at the same time.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Kristina Doskich