Move sidekiq_* metrics out of gitlab-exporter

We're looking to sunset gitlab-exporter in #288043 (closed).

NOTE: This epic used to suggest that we drop gitlab-exporter entirely. We have come to understand that this might not be the best solution or only option. The epic has been updated to reflect that.

There are a number of Sidekiq related metrics exported from it that we either need to retire or move:

• sidekiq_jobs_processed_total
• sidekiq_jobs_failed_total
• sidekiq_jobs_enqueued_size
• sidekiq_jobs_scheduled_size
• sidekiq_jobs_retry_size
• sidekiq_jobs_dead_size
• sidekiq_default_queue_latency_seconds
• sidekiq_processes_size
• sidekiq_workers_size
• sidekiq_queue_size
• sidekiq_queue_latency_seconds
• sidekiq_queue_paused
• sidekiq_enqueued_jobs
• sidekiq_running_jobs
• sidekiq_to_be_retried_jobs
• sidekiq_dead_jobs

The goal of this issue is to:

• Identify which of the mentioned metrics are still in use. If they are still in use, they should be documented. If not, they should be removed.
• If the metrics are in use, move them out of the exporter. This could happen in various ways, some suggestions below.

Use of the metrics

metric	exported from	replaced by	documented	used
sidekiq_concurrency	middleware	-	Y	Y
sidekiq_dead_jobs	gitlab-exporter	sidekiq_jobs_dead_total	N	N
sidekiq_dead_jobs_total	???	sidekiq_jobs_dead_total	N	Y (Thanos)
sidekiq_default_queue_latency_seconds	gitlab-exporter	sidekiq_jobs_queue_duration_seconds ?	N	N
sidekiq_elasticsearch_requests_duration_seconds	middleware	-	Y	Y
sidekiq_elasticsearch_requests_total	middleware	-	Y	Y
sidekiq_enqueued_jobs	gitlab-exporter	sidekiq_enqueued_jobs_total	N	N
sidekiq_enqueued_jobs_total	middleware	-	N	Y
sidekiq_jobs_completion_seconds	middleware	-	Y	Y
sidekiq_jobs_cpu_seconds	middleware	-	Y	Y
sidekiq_jobs_db_seconds	middleware	-	Y	Y
sidekiq_jobs_dead_size	gitlab-exporter	sidekiq_jobs_dead_total	N	N
sidekiq_jobs_dead_total	middleware	-	Y	Y
sidekiq_jobs_enqueued_size	gitlab-exporter	sidekiq_enqueued_jobs_total	N	N
sidekiq_jobs_failed_total	middleware	-	Y	Y
sidekiq_jobs_failed_total	gitlab-exporter	DUPLICATE	N	N
sidekiq_jobs_gitaly_seconds	middleware	-	Y	Y
sidekiq_jobs_processed_total	gitlab-exporter	-	N	N
sidekiq_jobs_queue_duration_seconds	middleware	-	Y	Y
sidekiq_jobs_retried_total	middleware	-	Y	Y
sidekiq_jobs_retry_size	gitlab-exporter	sidekiq_jobs_retried_total	N	N
sidekiq_jobs_scheduled_size	gitlab-exporter	-	N	N
sidekiq_processes_size	gitlab-exporter	-	N	N
sidekiq_queue_latency_seconds	gitlab-exporter	sidekiq_jobs_queue_duration_seconds	N	N
sidekiq_queue_paused	gitlab-exporter	-	N	N
sidekiq_queue_size	gitlab-exporter	-	N	Y (Thanos, RB)
sidekiq_redis_requests_duration_seconds	middleware	-	Y	Y
sidekiq_redis_requests_total	middleware	-	Y	Y
sidekiq_running_jobs	middleware	-	Y	Y
sidekiq_running_jobs	gitlab-exporter	DUPLICATE	N	N
sidekiq_to_be_retried_jobs	gitlab-exporter	-	N	N
sidekiq_workers_size	gitlab-exporter	-	N	N

Sheet

Migration of metrics

TODO