Put the job to dead queue if retry_count is exhausted
The jobs where retry_count is exhausted are rejected on Sidekiq level before the job is put to retry
queue. In other words, it happens when the job actually fails. If the job was killed, then we can only handle it when we return it back into the queue. So we increment retry_count
each time we pull abandoned job from the working queue and when retry_count
is exhausted we put the job to the Sidekiq dead
queue.
There are some trades off we have here:
- No progressive backoff time for retries
- We calculate
retry_count
when we search for abandoned jobs in working queues, which effectively means that we do that every hour (by default)
How we plan to deploy it:
After merging this we need to build the new gem and update it on GitLab side. For the worst-case scenario, we'll add a feature flag to be able to disable the reliable fetcher.
Edited by Valery Sizov