Deferred sidekiq jobs are deduplicated incorrectly for until_executed strategy
Summary
If a worker has deduplicate :until_executed, if_deduplicated: :reschedule_once, including_scheduled: true
options and if the worker is deferred
, the deferred retry would be deduplicated again ending up in a state where the deferred job not executing after the worker is enabled again.
This happened during an incident, where we deferred a worker that has until_executed
deduplication strategy and the worker's queue backlog was cleared due to the de-duplication as the deferred jobs are scheduled after 5 minutes
More context: gitlab-com/gl-infra/production#17282 (comment 1696882954)
Steps to reproduce
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)