Implement different deduplication strategies for idempotent jobs
In gitlab-org/gitlab!25447 (merged) we've added a single strategy for recognizing duplicate jobs. The strategy takes a lock when a job is added to the queue, and removes that lock before the job starts.
We could add more strategies for covering more duplicates similar to what unique-jobs provides.
Our implementation would need to take the idempotent? attribute of a worker into account before deduplicating, so we can't switch to using that gem just yet. But we could make the API for specifying a deduplication method on a worker compatible.
The API works by adding a lock: option to the sidekiq_options. If a worker specifies a lock explicitly, we could consider it idempotent?, if no lock is specified, we default to :until_executing.
The gem provides the following strategies:
-
:until_executing(detection implemented in gitlab-org/gitlab!25447 (merged)) Lock until the previously scheduled job starts. -
:until_executedLock until the previously scheduled job finishes -
:until_expiredLock solely based on time -
:until_and_while_executingPrevents a job from being scheduled twice, but allows a job to be scheduled when one was already running. Does not allow 2 jobs to run simultaneously. -
:while_executingAllows jobs to be scheduled at the same time, but does not allow them to be run simultaneously. This could be a replacement for ourexclusive_leasepattern. -
:noneSince we automatically deduplicate with:until_executingwe might want to add a:noneif a job should not be deduplicated, even though it is idempotent