Geo status unhealthy and out-of-date for 6 hours after Sidekiq was killed

Follow up #375298 (comment 1417017018):

I believe the cause of Geo::MetricsUpdateWorker getting deduplicated for 6 hours is because of "Performing with-downtime upgrades while Sidekiq jobs are being processed causes exceptions during jobs".

But I do think we should set Geo::MetricsUpdateWorker's ttl to 10 or 20 minutes.

The Sidekiq idempotency_key ttl should be set to longer than the longest expected duration of that job. I think 20 minutes fits. Also note that it is not necessarily dangerous for these Geo metrics jobs to overlap a bit.

If these jobs are taking 20+ minutes, it is an indication of either poor database performance, or massive scale which has revealed some part of this job to be not actually scalable.

We should also reasonably reduce the exclusive lease timeout to 20 minutes also. I assume it suffers from the same problem when exceptions occur.

Workaround

In Rails console in the site which is failing to run Geo::MetricsUpdateWorker due to Sidekiq deduplication:

worker_class = Geo::MetricsUpdateWorker
args = []
dj = Gitlab::SidekiqMiddleware::DuplicateJobs::DuplicateJob.new({ 'class' => worker_class.name, 'args' => args }, worker_class.queue)
dj.send(:idempotency_key)
dj.delete!

References:

Edited by Michael Kozono