Make the ProcessCommitWorker idempotent
Part of #178 (closed)
The ProcessCommitWorker
processed 1 million "duplicate" jobs the past 7 days, and spent 120 hours on that.
Duplicate jobs are jobs that get scheduled when there is already a job in the queue for the same worker with the same arguments.
If the job was (marked as) idempotent, we would be able to deduplicate those jobs when they get scheduled.
This job closes issues if the commit message contained a reference to close the issue and was pushed to the default branch. It does not do this if the commit was for a merged merge request. It also creates system notes and cross references when it closes issues.
It also creates metrics for referenced issues: If an issue is referenced, it updates the the issue-metrics and sets the committed-date as the first time the issue was referenced, if the previous date was newer than the current commit date.
This job runs for every commit pushed, but it does the same thing or nothing if it was already done, so it could be deduplicated.