Deduplicate PipelineProcessWorker and other SIdekiq jobs

From gitlab-com/gl-infra/production#1014 (closed), we had a peak of 15,000 Sidekiq jobs for PipelineProcessWorker.

As @ayufan mentioned, we don't de-duplicate jobs at the moment, so it's possible we're doing too much work.

We could conceivably create a custom Sidekiq fetcher for this queue that periodically scans all jobs and removes duplicates before fetching a job from the queue.

@ayufan @dosuken123 What other ways could we do this? (e.g. insert a database entry/hash that pipeline has finished, etc.)