Record sidekiq scheduling latency as a histogram

Spun out of https://gitlab.com/gitlab-org/gitlab-ce/issues/64649

Currently sidekiq scheduling latency is recorded as a counter. This needs to be changed to a histogram so that we can do adequate analysis on this metric.

https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/metrics/sidekiq_middleware.rb#L14-15

Current Metric: https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=sum(rate(gitlab_transaction_sidekiq_queue_duration_total%5B5m%5D))%20by%20(controller)&g0.tab=1

@bjk-gitlab suggested bucket sizes of [1, 5, 10, 50, 100, 500, 1000, 5000], I think this is fair.

Assignee Loading
Time tracking Loading