Add counter metric for deferred jobs

What does this MR do and why?

Related to #472976 (closed)

This MR adds a counter metric for every job that is deferred by the concurrency limit middleware

The MR also updates documentation and adds info about the available counters to the concurrency limit middleware documentation to help developers to discover them

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

patch for testing

diff --git a/ee/app/workers/zoekt/indexer_worker.rb b/ee/app/workers/zoekt/indexer_worker.rb
index 70bbeacd7d16..8fc3b42713eb 100644
--- a/ee/app/workers/zoekt/indexer_worker.rb
+++ b/ee/app/workers/zoekt/indexer_worker.rb
@@ -20,9 +20,10 @@ class IndexerWorker
     sidekiq_options retry: 2
     idempotent!
     pause_control :zoekt
-    concurrency_limit -> { 30 if Feature.enabled?(:zoekt_limit_indexing_concurrency) }
+    concurrency_limit -> { 1 }
 
     def perform(project_id, options = {})
+      sleep(30)
       return unless ::Gitlab::CurrentSettings.zoekt_indexing_enabled?
       return unless ::License.feature_available?(:zoekt_code_search)
 

1. enable sidekiq metrics in gdk

 gdk config set gitlab.rails_background_jobs.sidekiq_exporter_enabled true
 gdk reconfigure

2. apply the patch above to manually set this worker very low, and add a sleep timer to cause the worker queue to back up
3. in rails console, queue the worker for every project in your database

    Project.all.each { |p| Zoekt::IndexerWorker.perform_async(p.id) }

4. load the metrics page for sidekiq metrics: http://gdk.test:3807/metrics
5. ensure the values are updated for the metric

note: you might need to gdk restart rails-background-jobs to make the metrics show up

# HELP sidekiq_concurrency_limit_deferred_jobs_total Multiprocess metric
# TYPE sidekiq_concurrency_limit_deferred_jobs_total counter
sidekiq_concurrency_limit_deferred_jobs_total{worker="Zoekt::IndexerWorker"} 64
# HELP sidekiq_concurrency_limit_max_concurrent_jobs Multiprocess metric
# TYPE sidekiq_concurrency_limit_max_concurrent_jobs gauge
sidekiq_concurrency_limit_max_concurrent_jobs{worker="ElasticCommitIndexerWorker",pid="sidekiq_0"} 30
sidekiq_concurrency_limit_max_concurrent_jobs{worker="PackageMetadata::GlobalAdvisoryScanWorker",pid="sidekiq_0"} 10
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Search::Zoekt::DeleteProjectWorker",pid="sidekiq_0"} 100
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanExecutionPolicies::CreatePipelineWorker",pid="sidekiq_0"} 10000
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Security::ScanResultPolicies::AddApproversToRulesWorker",pid="sidekiq_0"} 200
sidekiq_concurrency_limit_max_concurrent_jobs{worker="Zoekt::IndexerWorker",pid="sidekiq_0"} 1
# HELP sidekiq_concurrency_limit_queue_jobs Multiprocess metric
# TYPE sidekiq_concurrency_limit_queue_jobs gauge
sidekiq_concurrency_limit_queue_jobs{worker="ElasticCommitIndexerWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="PackageMetadata::GlobalAdvisoryScanWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Search::Zoekt::DeleteProjectWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Security::ScanExecutionPolicies::CreatePipelineWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Security::ScanResultPolicies::AddApproversToRulesWorker"} 0
sidekiq_concurrency_limit_queue_jobs{worker="Zoekt::IndexerWorker"} 24

doc/administration/monitoring/prometheus/gitlab_metrics.md

@@ -234,6 +234,7 @@ configuration option in `gitlab.yml`. These metrics are served from the
 | `sidekiq_mem_total_bytes`                      | Gauge     | 15.3 | Number of bytes allocated for both objects consuming an object slot and objects that required a malloc'|                                                                   |
 | `sidekiq_concurrency_limit_queue_jobs`         | Gauge     | 17.3 | Number of Sidekiq jobs waiting in the concurrency limit queue|  `worker`                                                             |
 | `sidekiq_concurrency_limit_max_concurrent_jobs` | Gauge     | 17.3 | Max number of concurrent running Sidekiq jobs |   `worker`                                                           |
+| `sidekiq_concurrency_limit_deferred_jobs_total` | Counter     | 17.3 | Total number of deferred Sidekiq jobs |   `worker`                                                           |
 | `geo_db_replication_lag_seconds`               | Gauge   | 10.2  | Database replication lag (seconds) | `url` |
 | `geo_repositories`                             | Gauge   | 10.2  | Total number of repositories available on primary | `url` |
 | `geo_lfs_objects`                              | Gauge   | 10.2  | Number of LFS objects on primary | `url` |