Refactor Reenqueuer to not sleep
The following discussion from !22031 (merged) should be addressed:
-
@mikolaj_wawrzyniak started a discussion: (+4 comments) I'm still not convinced for usage of
sleep
here. It feel wasteful, to put process to sleep. I wonder maybe we can try to uselease
mechanism here as well.
Possible solution
- Reenqueue at a specific time immediately after the end of minimum duration.
- Take another exclusive lease, open-ended, which expires exactly at the end of minimum duration.
Note:
Sidekiq scheduled jobs introduces another delay that averages to the poll interval.
Checking for New Jobs
Sidekiq's scheduler is not meant to be second-precise. It checks for scheduled jobs approximately every 5 seconds by default (15 seconds before Sidekiq 5.1)
A case where this matters:
- There is a ton of work to do (backfill condition), therefore we don't really want to insert a lot of delay between jobs.
- The job is performant, and even with maximum work to do, it always completes below minimum duration. Great!
If we ignore poll interval mechanics, we might think that with a minimum interval of 5s, the job will run every ~5s. But with a default poll interval of 5s, it would actually run closer to every 10s.
Therefore we should subtract average_scheduled_poll_interval
from minimum_interval. If the wait duration is < 0, then reenqueue without a target time.