Sidekiq duplicate entries created for jobs in the scheduled set
When you use perform_at
to run a Sidekiq job in the future, it doesn't go to the named queue, it goes to the scheduled set: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/scheduled.rb
However, we still set a duplicate-of
key if the job was a duplicate of a job that exists in the queue when it was scheduled. So if I did:
AuthorizedProjectsWorker.perform_async(1)
AuthorizedProjectsWorker.perform_at(Time.now + 1.day, 1)
Then the second job would be marked as a duplicate of the first, even though it probably shouldn't be as we intended to do something at a specific time. It doesn't look like client middleware applies when moving a job from the scheduled set into its 'regular' queue, so I'm not sure exactly what the best solution is here - maybe we should just exempt future jobs entirely?
This also applies in a slightly different way to retries. If a job is retried, it will go into the retry set, and similarly we should probably not deduplicate it at that point?