Ensure all jobs can handle duplication

It's happened in more than one production incident that jobs were duplicated, causing duplicate emails, duplicate system notes on MRs, etc, which looks bad. An incident happened recently that caused lots of job duplication. It is possible there are other consequences of duplicate jobs that we are unaware of.

But, job duplication is explicitly allowed to occur in Sidekiq:

• https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/manager.rb#L121-L126
• https://github.com/mperham/sidekiq/wiki/FAQ#what-happens-to-long-running-jobs-when-sidekiq-restarts

Sidekiq only tries to guarantee at-least-once execution because it's too hard to guarantee exactly-once execution, along with scalability and efficiency. So the contract is: all jobs must be able to handle duplication.

Idempotent jobs solution

• Open issue to document idempotency requirement in https://docs.gitlab.com/ee/development/sidekiq_style_guide.html
• Open issue to look through all our jobs for violations and open issues to fix them
• Open issue to help devs maintain this contract in GitLab code
  • E.g. Add IDEMPOTENT = true to all workers, require it in tests or just in Rubocop, and add an explanation or a link to the doc in the test failure. Note, it's ok to be flexible: IDEMPOTENT = 'No, but we accept that emails may be delivered more than once.'.

Deduplicate jobs solution

• Exclusive lease on JID?