Skip to content

Use 1.week TTL for importer JobWaiter keys

Luke Duncalfe requested to merge 422976-increase-job-waiter-ttl into master

What does this MR do and why?

This MR extends the JobWaiter key expiry time for importers to 1 week.

Click to read a description of `JobWaiter` and how it's used in importers.
  1. Our importers will queue a number of workers to process data fetched from APIs like GitHub. For each worker queued we increment a JobWaiter counter to track the count of jobs queued.
  2. When one of the queued worker completes, it calls JobWaiter.notify to report that it has finished. This pushes its Sidekiq Job ID (jid) to a Redis set. (What's pushed isn't important but rather just that 1 of something was pushed to the set).
  3. Gitlab::GithubImport::AdvanceStageWorker is queued with the count of jobs queued. It calls JobWaiter.wait to clear the redis set and decrement the count. It does this periodically until the count is 0 (when all jobs have been notified that they completed).

In very large imports sometimes more than 6 hours (the current key expiry time) pass between when the last job of a certain batch of jobs notifies (calls JobWaiter.notify), and when we read the job waiter set (call JobWaiter.wait).

This results in these imports becoming stuck, as they continue to wait for the expired notifications.

Extending this time to 7 days will cover very large imports. Shorter imports will continue to pop data from the set as soon as they call JobWaiter.wait, so for non-stuck imports (the vast majority) this change won't increase the size of data stored in Redis.

#422976 (closed)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #422976 (closed)

Merge request reports